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Why GAO Did This Study 

To enhance aviation security, the 
Transportation Security 
Administration (TSA) began initial 
testing in October 2003 of its 
Screening of Passengers by 
Observation Techniques (SPOT) 
program. Behavior Detection 
Officers (BDO) carry out SPOT's 
mission to identify persons who 
pose a risk to aviation security by 
focusing on behavioral and 
appearance indicators. GAO was 
asked to review the SPOT program. 
GAO analyzed (1) the extent to 
which TSA validated the SPOT 
program before deployment, (2) 
implementation challenges, and (3) 
the extent to which TSA measures 
SPOT's effect on aviation security. 
GAO analyzed TSA documents, 
such as strategic plans and 
operating procedures; interviewed 
agency personnel and subject 
matter experts; and visited 15 
SPOT airports, among other things. 
Although the results from these 
visits are not generalizable, they 
provided insights into SPOT 
operations. 



What GAO Recommends 



GAO recommends that TSA, among 
other things, use an independent 
panel of experts to assist in 
validating SPOT, enhance SPOT 
data collection and analysis, fully 
utilize TSA resources to identify 
possible threats, and establish a 
plan to develop more outcome- 
oriented measures for SPOT. DHS 
reviewed a draft of this report and 
generally concurred with our 
recommendations although its 
plans do not fully address one of 
our recommendations. 



View GAO-1 0-763 or key components. 
For more information, contact Stephen M. 
Lord at (202) 512-4379 or lords@gao.gov. 



What GAO Found 

Although the Department of Homeland Security (DHS) is in the process of 
validating some aspects of the SPOT program, TSA deployed SPOT 
nationwide without first validating the scientific basis for identifying 
suspicious passengers in an airport environment. A scientific consensus does 
not exist on whether behavior detection principles can be reliably used for 
counterterrorism purposes, according to the National Research Council of the 
National Academy of Sciences. According to TSA, no other large-scale 
security screening program based on behavioral indicators has ever been 
rigorously scientifically validated. DHS plans to review aspects of SPOT, such 
as whether the program is more effective at identifying threats than random 
screening. Nonetheless, DHS's current plan to assess SPOT is not designed to 
fully validate whether behavior detection can be used to reliably identify 
individuals in an airport environment who pose a security risk. For example, 
factors such as the length of time BDOs can observe passengers without 
becoming fatigued are not part of the plan and could provide additional 
information on the extent to which SPOT can be effectively implemented. 
Prior GAO work has found that independent expert review panels can provide 
comprehensive, objective reviews of complex issues. Use of such a panel to 
review DHS's methodology could help ensure a rigorous, scientific validation 
of SPOT, helping provide more assurance that SPOT is fulfilling its mission to 
strengthen aviation security. 

TSA is experiencing implementation challenges, including not fully utilizing 
the resources it has available to systematically collect and analyze the 
information obtained by BDOs on passengers who may pose a threat to the 
aviation system. TSA's Transportation System Operations Center has the 
resources to investigate aviation threats but generally does not check all law 
enforcement and intelligence databases available to it to identify persons 
referred by BDOs. Utilizing existing resources would enhance TSA's ability to 
quickly verify passenger identity and could help TSA to more reliably "connect 
the dots." Further, most BDOs lack a mechanism to input data on suspicious 
passengers into a database used by TSA analysts and also lack a means to 
obtain information from the Transportation System Operations Center on a 
timely basis. TSA states that it is in the process of providing input 
capabilities, but does not have a time frame for when this will occur at all 
SPOT airports. Providing BDOs, or other TSA personnel, with these 
capabilities could help TSA "connect the dots" to identify potential threats. 

Although TSA has some performance measures related to SPOT, it lacks 
outcome-oriented measures to evaluate the program's progress toward 
reaching its goals. Establishing a plan to develop these measures could better 
position TSA to determine if SPOT is contributing to TSA's strategic goals for 
aviation security. TSA is planning to enhance its evaluation capabilities in 
2010 to more readily assess the program's effectiveness by conducting 
statistical analysis of data related to SPOT referrals to law enforcement and 
associated arrests. 
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United States Government Accountability Office 
Washington, DC 20548 



May 20, 2010 

The Honorable John L. Mica 
Ranking Member 

Committee on Transportation and Infrastructure 
House of Representatives 

Dear Mr. Mica: 

The terrorist attacks of September 11, 2001, highlighted the need to 
improve security within the nation's civil aviation system to deter persons 
seeking to repeat similar attacks on the nation's critical infrastructure. In 
October 2003, the Transportation Security Administration (TSA) of the 
Department of Homeland Security (DHS) conducted an operational test of 
the use of behavior detection techniques to screen passengers in an 
airport environment, and subsequently began training certain 
Transportation Security Officers (TSO) — TSA employees responsible for 
screening passengers and their property — in these techniques. These 
TSOs performed behavior observation as a collateral duty. Beginning in 
fiscal year 2007, TSA created separate Behavior Detection Officer (BDO) 
positions as part of the Screening of Passengers by Observation 
Techniques (SPOT) program. 1 According to TSA, the SPOT program is a 
derivative of other behavioral analysis programs that have been 
successfully employed by law enforcement and security personnel both in 
the United States and around the world, particularly that of Israel's airline, 
E1A1. 2 

TSA designed SPOT to provide BDOs with a means of identifying persons 
who may pose a potential security risk at TSA-regulated airports 3 by 
focusing on behaviors and appearances that deviate from an established 



'BDOs must have at least 12 months experience as a TSO, or related security work 
experience, and must pass a BDO training course. 

2 TSA cautions that the applicability of El Al's security processes to those used by TSA is 
constrained by differences in the scale of El Al's worldwide operations and the flexibilities 
that El Al has in implementing security processes compared to constraints on TSA. For 
example, El Al security screeners are encouraged to spend as much time with passengers 
as needed, and are not concerned whether passengers experience delays in boarding an 
aircraft. 

3 For the purposes of this report, the term "TSA-regulated airport" refers to a U.S. airport 
operating under a TSA-approved security program. 
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baseline, and that may be indicative of stress, fear, or deception. 
Passengers in an airport terminal, including those waiting in security 
checkpoint lines, are observed by the BDOs to determine if their 
behavioral and appearance indicators — which are assigned varying points 
by SPOT — have (in combination) exceeded a predetermined numerical 
threshold. In cases where the passenger exceeds the threshold, the 
passenger is referred for additional screening by BDOs and a TSO. During 
this referral screening, if the passenger exhibits behaviors that exceed 
another numerical threshold, they are to be referred to a law enforcement 
officer (LEO) for further investigation. In addition to observing 
passengers at airport checkpoints, BDOs may patrol throughout an airport 
terminal, and sometimes participate in other activities, such as TSA's 
Visible Intermodal Prevention and Response team operations. These 
teams are responsible for periodically augmenting security at air and 
ground transportation facilities around the country. 4 

As of March 2010, TSA deployed about 3,000 BDOs at an annual cost of 
about $212 million; this force increased almost fifteen-fold between March 
2007 and July 2009. BDOs have been selectively deployed to 161 of the 457 
TSA-regulated airports in the United States at which passengers and their 
property are subject to TSA-mandated screening procedures. 5 The 
conference report accompanying the fiscal year 2010 DHS appropriations 
act provided that $211.9 million of aviation security funding was for the 
SPOT program. 6 The administration has requested $232 million for SPOT 
for fiscal year 2011, a $20.2 million (9.5 percent) increase over the current 
funding level. This increase would support a workforce increase from 
about 3,000 to 3,350 BDOs. If this funding request is approved and 
maintained, SPOT would cost about $1.2 billion over the next 5 years. 



4 Visible Intermodal Prevention and Response teams are comprised of federal air marshals, 
surface transportation security inspectors, TSOs, BDOs, and canines. 

5 TSA classifies its regulated airports in the United States into one of five categories — X, I, 
II, III, and IV. Generally, category X airports have the largest number of passenger 
boardings and category IV airports have the least. 

r 'See H.R. Rep. No. 111-298 at 77 (2009) (Conf. Rep.). The conference report directed TSA to 
report, no later than 60 days after enactment, on the scientific basis for using behavior 
pattern recognition for observing airline passengers for signs of hostile intent, the 
effectiveness of the SPOT program in meeting its goals and objectives, and the justification 
for expanding the program. The conference report also directed us to review this report 
and to provide our findings to the Committees no later than 120 days after the TSA report is 
submitted. TSA completed its report to Congress on March 15, 2010. 
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You asked us to address SPOT's development and implementation. This 
report addresses the following questions: 

1. To what extent did TSA determine whether SPOT had a scientifically 
validated basis for identifying passengers before deploying it and 
utilize recognized best practices during SPOT's development? 

2. What management challenges, if any, have emerged during the 
implementation of SPOT at the nation's airports? 

3. To what extent has TSA measured SPOT's effect on aviation security? 

4. To what extent has TSA incorporated the attributes of an effective 
training program into the training for SPOT? 

This report is a public version of the restricted report (GAO-10-157SU) 
that we provided to you on May 14, 2010. DHS and TSA deemed some of 
the information in the restricted report as sensitive security information, 
which must be protected from public disclosure. Therefore, this report 
omits this information. Although the information provided in this report is 
more limited in scope, it addresses the same questions as the restricted 
report. Also, the overall methodology used for both reports is the same. 

To determine the extent to which TSA determined whether SPOT had a 
scientifically validated basis for identifying passengers who may pose a 
risk to aviation security before deploying it, we reviewed literature on 
behavior analysis by subject matter experts, and analyzed relevant reports 
and books on the topic. These included a 2008 study by the National 
Research Council of the National Academy of Sciences that included a 
discussion section on deception and behavioral surveillance, as well as 
other issues related to behavioral analysis. 7 We interviewed seven 
recognized experts in the field, and an expert on emergency responses to 
terror attacks and mathematical models in operations management. 8 
Although the views of these experts cannot be generalized across all 
experts on behavior analysis, because we selected these individuals based 
on their publications on behavioral analysis or related topics, their 
recognized accomplishments and expertise, and, in some cases, TSA's use 
of their work or expertise to design and review the SPOT program's 



7 National Research Council, Protecting Individual Privacy in the Struggle Against 
Terrorists: A Framework for Assessment (Washington, D.C.: National Academies Press, 
2008). We reviewed the approach used and the information provided in this study and 
found the study and its results to be reliable for the purposes for which we used it in this 
report. 

8 See app. I for additional information on the experts we interviewed. 
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behaviors, they provided us with an understanding of the fundamentals of 
behavior analysis, and its use in airports. We also interviewed cognizant 
officials from other U.S. government agencies that utilize behavior analysis 
in their work, including U.S. Customs and Border Protection (CBP), the 
U.S. Secret Service, the Federal Air Marshall Service (FAMS), and the 
Federal Bureau of Investigation (FBI). 9 To better understand how SPOT 
incorporated expertise on behavior analysis for aviation security, we also 
interviewed current and retired officials of Israel's El Al Airlines, whose 
security processes TSA cites as providing part of the basis of the SPOT 
program. 10 

To determine to what extent TSA utilized best practices during SPOT's 
development — including carrying out a comprehensive risk assessment, a 
cost-benefit analysis, and a strategic plan — we interviewed program 
officials and reviewed related program documentation, including briefings 
used in the course of developing and fielding SPOT, strategic plans, and 
standard operating procedures. 11 We compared these documents to DHS's 
2006 Cost Benefit Analysis Guidebook, 12 Office of Management and 



9 For reasons of scope, we did not assess the scientific basis of the methods and processes 
used by these agencies in their application of behavioral detection. 

'"Although SPOT is based in some respects on El Al's aviation security program, El Al's 
processes differ in substantive ways from those used by the SPOT program. In particular, 
El Al does not use a list of specific behaviors with numerical values for each, or a 
numerical threshold to determine whether or not to question a passenger; rather, El Al 
security officers utilize behavioral indicators as a basis for interviewing all passengers 
boarding El Al passenger aircraft, and accessing relevant intelligence databases, when 
deemed appropriate. In addition, El Al officials told us that they train all their personnel — 
not just security officers — in elements of behavior analysis, and conduct covert tests of 
their employees' attentiveness at frequent intervals. According to these officials, El Al also 
permits what is termed "profiling," in which passengers may be singled out for further 
questioning based on their nationality, ethnicity, religion, appearance, or other ascriptive 
characteristics, but these are not the only basis on which a passenger may be questioned. 
In addition, El Al security officers are empowered to bar any passenger from boarding an 
aircraft. The scale of El Al operations is considerably smaller than that of major airlines 
operating within the United States. As of 2008, El Al had a fleet of 34 aircraft. In Israel, El 
Al operates out of one hub airport, Ben-Gurion International, and also flies to Eilat, a city in 
southern Israel; in contrast, there are 457 TSA-regulated airports in the United States. In 
2008, El Al had passenger boardings of about 3.6 million; in contrast, Southwest Airlines 
alone flew about 102 million passengers in the same year. 

"Unless otherwise noted in the report, we refer to the SPOT strategic plan issued in March 
2007. 

12 DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: Feb. 1, 2006). 
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Budget (OMB) guidance, 13 and DHS's 2006 and 2009 National 
Infrastructure Protection Plans (NIPP) , which set forth a risk management 
framework to guide security decision making and resource allocation 
decisions, and our previous work on the characteristics of an effective 
strategic plan. 

To identify any challenges that emerged during implementation of the 
SPOT program, we conducted field site visits to 15 TSA-regulated airports 
with SPOT that represent almost 10 percent of the 161 TSA-regulated 
airports with SPOT to observe operations and meet with key program 
personnel. 14 We chose airports with high, medium, and low passenger 
volume; airports with BDOs who are TSA (i.e., government) employees 
and an airport with BDOs employed by contractors as part of the TSA 
Screening Partnership Program; and airports with LEOs who were 
identified by TSA as having received some form of behavior detection 
training and airports where they were not known to have received such 
training. 15 We also selected airports on the basis of TSA's assessment of 
which ones are at highest risk of attack by terrorists, including the 2 that 
ranked the highest, as reported in TSA's Current Airport Threat 
Assessment. 16 Since the airports we selected range broadly in terms of 
passenger volume, physical size and layout, geographic location, and 
potential value as a target for terrorism, among other things, the results 
from these visits are not generalizable to other airports. However, these 
visits provided helpful insights into the operation of SPOT at airports. In 
addition, to determine whether challenges emerged in implementing 
SPOT, we compared TSA's approach for implementing and managing 



13 OMB, Circular No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of 
Federal Programs (Washington, D.C.: October 1992); Circular No. A-4, Regulatory 
Analysis (Washington, D.C.: Sept. 2003). 

1J See app. I for additional details on the airports we visited. 

15 At airports participating in TSA's Screening Partnership Program, private-sector 
contractors perform screening activities, including SPOT, in accordance with TSA 
requirements and oversight. See 49 U.S.C. § 44920. Unless otherwise specified, references 
to TSOs include private-sector contract screeners. For more information, see GAO, 
Aviation Security: Progress Made to Set Up Program Using Private-Sector Airport 
Screeners, but More Work Remains, GAO-06-166 (Washington, D.C.: Mar. 31, 2006). 

"The TSA Current Airport Threat Assessment is a threat estimate designed to provide a 
snapshot of the current terrorist threat to airports in the United States as well as for major 
international airports serving as last points of departure for U.S. airlines. 
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SPOT to our Standards for Internal Control in the Federal Government 11 
and to risk management principles we had previously identified. 18 In 
reviewing TSA's approach to developing and implementing SPOT, we 
considered relevant laws, regulations, and other materials, including those 
related to privacy, such as TSA's Privacy Impact Assessments. To obtain 
comparative data on how SPOT had been implemented at different 
airports across the nation, we conducted a survey of all Federal Security 
Directors responsible for security operations at TSA-regulated airports 
with SPOT. 19 (This accounted for all 161 TSA-regulated airports with 
SPOT because a single Federal Security Director may be responsible for 
several airports.) We obtained a 100 percent response rate. This survey 
asked, among other things, about the relationship between LEOs and the 
airport authority and BDOs. In addition, to understand the interaction of 
BDOs and LEOs, as well as other SPOT implementation issues, at each of 
the 15 TSA-regulated airports we visited we spoke with BDO managers, 
Federal Security Directors, Assistant Federal Security Directors, 1 or 2 
BDOs, and 1 or 2 LEOs. 

To determine the extent to which TSA has measured SPOT's effect on 
aviation security, we obtained and analyzed the TSA SPOT referral 
database, 20 which aims to record all incidents in which passengers who 
have passed through the checkpoint are sent to SPOT referral screening 
for additional questioning and screening of property and person. The 
database also maintains records of instances where passengers were 
referred by a BDO to a LEO for questioning. We assessed the reliability of 
the SPOT referral data by (1) performing electronic testing of required 
data elements, (2) reviewing existing information about the data and the 



17 GAO, Standards for Internal Controls in the Federal Government, GAO/AIMD-00-21.3.1 
(Washington, D.C.: November 1999). 

ls See GAO, Risk Management: Further Refinements Needed to Assess Risks and 
Prioritize Protective Measures at Ports and Other Critical Infrastructure, GAO-06-91 
(Washington, D.C.: Dec. 15, 2005) and Transportation Security: Comprehensive Risk 
Assessments and Stronger Internal Controls Needed to Help Inform TSA Resource 
Allocation, GAO-09-492 (Washington, D.C.: Mar. 27, 2009). 

'"Federal Security Directors are the highest ranking TSA officials responsible for security 
operations at TSA-regulated airports. See 49 U.S.C. § 44933. They and their assistants 
coordinate with both federal and nonfederal entities present at their airports, including the 
FAMS, the Drag Enforcement Administration, and CBP. When appropriate, Federal 
Security Directors may bar an individual from boarding an aircraft. 

2 °The SPOT referral data we analyzed covered the period May 29, 2004, through August 31, 
2008. These were the data available at the time of our analysis. 
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system that produced them, and (3) interviewing agency officials 
knowledgeable about the data. We found a number of problems related to 
how the data were collected and recorded that are discussed later in this 
report. As a result, we were unable to use the SPOT referral data to assess 
whether any behavior or combination of SPOT behaviors could be used to 
reliably predict the final outcome of an incident involving the use of SPOT. 
However, with the stated limitations in mind, and after resolving certain 
contradictions and anomalies in the database, we utilized the SPOT 
referral data to provide examples of information used by TSA to report on 
the program's performance, including a count of arrests and the reasons 
for those arrests. In addition, to determine if individuals who were later 
charged with or pleaded guilty to terrorism-related offenses had transited 
SPOT airports and whether TSA could obtain information from these 
transits to enhance its understanding of terrorist behaviors, we reviewed 
CBP and Department of Justice information to (1) identify individuals who 
were charged with or pleaded guilty to terrorism-related offenses and (2) 
determine if these individuals had, prior to being charged, transited 
airports where SPOT had been deployed. Further, we used our survey of 
Federal Security Directors at SPOT airports to determine the extent to 
which video surveillance cameras, which could make video recordings of 
terrorists transiting airports, are present at checkpoints. 

To assess the extent that SPOT training incorporates the attributes of an 
effective training program, we had TSA training experts complete a 
training assessment tool that we developed using guidance we prepared in 
our previous work for assessing training courses and curricula. 21 To better 
understand how other entities train their employees in behavior detection, 
and what their curricula include, we conducted site visits to the Secret 
Service, CBP, FAMS, and the FBI, and also interviewed nongovernmental 
experts on aspects of behavior detection training. We interviewed BDOs 
and BDO managers about the SPOT training. In addition, we interviewed 
El Al officials with regard to how El Al trains and tests its personnel in 
behavior recognition and analysis. Appendix I contains additional details 
about our scope and methodology. 

We conducted this performance audit from May 2008 through May 2010 in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 



21 GAO, Human Capital: A Guide for Assessing Strategic Training and Development 
Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: Mar. 1, 2004). 
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sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 



Background ^ e Aviation and Transportation Security Act established TSA as the 

° federal agency with primary responsibility for securing the nation's civil 

aviation system, which includes the screening of all passenger and 
property transported by commercial passenger aircraft. 22 TSA currently 
has direct responsibility for, or oversees the performance of, security 
operations at approximately 457 TSA-regulated airports in the United 
States implementing security requirements in accordance with TSA- 
approved security programs and other TSA direction. 23 At TSA-regulated 
airports, prior to boarding an aircraft, all passengers, their accessible 
property, and their checked baggage are screened pursuant to TSA- 
established procedures, which include, for example, passengers passing 
through security checkpoints where they and their identification 
documents are checked by TSOs and Travel Document Checkers, or by 
Screening Partnership Program employees. 



TSA uses multiple layers of security to deter, detect, and disrupt persons 
posing a potential risk to aviation security. These layers include three 
principal types of screening employees at airport checkpoints — Travel 
Document Checkers, who examine tickets, passports, and other forms of 
identification; TSOs, who examine property, including checked baggage, 
and persons using x-ray equipment and magnetometers, as well as other 
devices; and BDOs, using SPOT to assess passenger behaviors and 
appearance. 24 BDOs are the only type of TSA screening employees not 
deployed to all TSA-regulated airports and all checkpoints within the 



22 See Pub. L. No. 107-71, 115 Stat. 597 (2001). For purposes of this report, "commercial 
aircraft" refers to a U.S. or foreign-based air carrier operating under TSA-approved security 
programs with regularly scheduled passenger operations to or from a U.S. airport. 

23 See 49 C.F.R. pt. 1542. Some commercial airports with fewer than 2,500 annual 
enplanements (passengers boarding an aircraft) do not have TSA-approved screening 
processes. Enplanements are the number of paying passengers on a scheduled or 
nonscheduled (charter) flight. Infants and airline personnel are not included. A stop at an 
airport is not considered an enplanement if the passenger does not transfer aircraft. 

24 Private-sector screeners under contract to and overseen by TSA, and not TSOs, perform 
screening activities at airports participating in TSA's Screening Partnership Program. See 
49 U.S.C. § 44920. 
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airports where it is deployed on a regular basis. TSA deployed SPOT as an 
added layer of security to help deter terrorists attempting to exploit TSA's 
focus on prohibited items and other potential security weaknesses. Other 
security layers cited by TSA include intelligence gathering and analysis; 
passenger prescreening; random canine team searches at airports; federal 
air marshals; reinforced cockpit doors; federal flight deck officers; the 
passengers themselves; as well as other measures both visible and 
invisible to the public. Figure 1 shows TSA's 20 aviation security layers. 



Figure 1 : TSA's Layers of Aviation Security 



20 layers of security 




Source: TSA. 

"The No-Fly List is used to identify individuals who should be prevented from boarding an aircraft; it 
contains applicable records from the FBI's Terrorist Screening Center consolidated database of 
known or suspected terrorists. 

"The four layers inside the grey bar are screening layers of security applied to passengers and their 
property. 
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The grey area in figure 1 highlights four layers that apply to passengers 
and their property as they seek to board an aircraft. Airport LEOs, another 
layer of security cited by TSA, do not report to TSA and may not maintain 
a physical presence at smaller TSA-regulated airports. According to TSA, 
each one of these layers alone is capable of stopping a terrorist attack. In 
combination, TSA states that their security value is multiplied, creating a 
much stronger system, and that a terrorist who has to overcome multiple 
security layers in order to carry out an attack is more likely to be 
preempted, deterred, or to fail during the attempt. 



SPOT Uses Behavior 
Detection Techniques to 
Assess Passenger 
Behaviors and 
Appearances 



The SPOT program utilizes behavior observation and analysis techniques 
to identify potentially high-risk passengers. Individuals who exhibit 
suspicious behaviors, including both physical and appearance indicators, 
may be required to undergo additional screening. Field agents and law 
enforcement officers of other federal agencies and entities — such as the 
FBI, the Secret Service, CBP, and FAMS — utilize elements of behavior 
detection analysis as a part of their work. In addition, some foreign 
entities, such as Israel's El Al airlines, use behavior detection and analysis 
techniques as part of their security efforts. However, TSA emphasized to 
us that the SPOT program is unique among these entities because it uses a 
point system to help identify suspicious persons on the basis of their 
behavior and appearance and because behavior detection and analysis are 
the central focus of SPOT. Officials from the other agencies stated that 
their field personnel incorporate behavior detection as one of many skills 
used in their work; in contrast, behavior detection is the primary element 
of the BDOs' work. 



SPOT trains BDOs to look for and recognize facial expressions, body 
language, and appearance that indicate the possibility that an individual is 
engaged in some form of deception and fears discovery. These behaviors 
and appearances are listed on a SPOT score sheet used in SPOT training. 

Passenger behavior and appearance are to be compared by the BDOs — 
who typically work in two-person teams. BDOs are expected to "walk the 
line" — that is, to initiate casual conversations with passengers waiting in 
line, particularly if their observations led them to question someone 
exhibiting behaviors or appearances on the SPOT checklist. As the BDOs 
walk the line, and the passenger with SPOT indicators is reached, a casual 
conversation is used to determine if there is a basis for observed behaviors 
or appearances on the checklist. In most instances, these conversations 
provide information to the BDOs that permits them to consider the issue 
resolved, and hence not a security concern. Figure 2 below illustrates the 
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first step of the three-step SPOT process, the BDO-passenger interaction at 
a checkpoint prior to the passenger passing through a magnetometer. 



Figure 2: The First Step in the SPOT Process: BDOs Observing Passengers About to Go Through Checkpoint Magnetometer 




Step 1 


1. BDOs scan the passengers 
in line and occasionally initiate 
casual conversation 


2. BDOs identify person(s) who 

exhibit clusters of suspicious behaviors 
that meet a given threshold 


3. BDOs identify passengers exhibiting 
behaviors that exceed SPOT numerical 
threshold for referral questioning 



Sources: GAO (analysis), ArtExplosion (clip art), TSA (data). 

Note: Circle around passenger shows a person who is exhibiting a cluster of suspicious behaviors. 
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As shown in figure 2, passenger behavior and appearance are observed by 
the BDOs as passengers wait in line for screening at a security checkpoint. 
Even if the checkpoint is busy, the BDOs must attempt to visually scan all 
the passengers waiting in line, as well as persons near the checkpoint, to 
determine if any are showing behaviors or appearances on the SPOT 
checklist. According to TSA, on average a BDO has approximately 30 
seconds to assess each passenger while the passenger waits in line. For 
passengers exhibiting indicators above baseline conditions, the BDOs are 
to (mentally) add up the points assigned to each indicator they observe. 
Both BDO team members must agree that observed indicators have 
exceeded the predetermined numerical threshold, although they do not 
have to identify the same indicators the passenger exhibited. In instances 
when a passenger's SPOT indicators place them above the numerical 
threshold, and the passenger has placed their property on the conveyor 
belt for x-raying, and has walked through the magnetometer or equivalent 
screening device for passengers, he or she will be directed to the second 
step of SPOT, referral screening. This involves additional questioning and 
physical search of their person and property by BDOs and TSOs. This 
referral screening occurs in the checkpoint area. 

If the passenger's behavior escalates further — accumulating more points 
based on the SPOT checklist — the BDOs are to refer the passenger to a 
LEO. A referral to a LEO is a potential third step in the SPOT process. 
BDOs are not LEOs — they do not conduct criminal investigations, carry 
weapons, or make arrests. 

After a passenger has been referred by the BDOs to a LEO, the LEO is then 
expected to independently determine, through additional investigation, 
such as questioning the passenger and, if appropriate, by conducting an 
identity verification and background check through the FBI's National 
Crime Information Center (NCIC), whether sufficient grounds exist to take 
further action, such as detaining or arresting the passenger. TSA officials 
who are LEOs also have access to NCIC, such as an airport's Assistant 
Federal Security Director for Law Enforcement or federal air marshals. 
NCIC is the FBI's computerized index of criminal justice information (i.e., 
criminal record history information, fugitives, stolen properties, and 
missing persons), available to federal, state, and local law enforcement 
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and other criminal justice agencies at all times. 25 Similarly, other federal 
LEOs also have such access, including CBP, and Drug Enforcement 
Agency (DEA) personnel. However, since both local and federal LEOs 
have other responsibilities, and may not be present at each operating 
checkpoint, BDOs may have to seek them out to request an NCIC check. 
According to TSA, aside from requiring that an airport maintain a law 
enforcement presence, 26 it exercises no jurisdiction over the law 
enforcement activities of non-TSA officers or entities at an airport; thus, it 
cannot require LEOs to conduct an NCIC check or to provide BDOs with 
information about the ultimate disposition of cases referred by them to 
LEOs. 

Once the LEO concludes his or her investigation and determines whether 
the passenger will be arrested or detained, TSA officials are to evaluate the 
security concerns to determine whether to allow the passenger to proceed 
to the boarding gate. (In some instances, a LEO might choose not to arrest 
or detain a passenger; TSA would then decide whether the infraction was 
sufficiently serious to necessitate barring the passenger from boarding.) 
After a referral incident has been resolved, BDOs are to enter information 
about the incident into TSA's SPOT referral database. The data entered 
are to include time, date, location of the incident, behaviors witnessed, 
prohibited items found (if any), and information on the LEO's response (if 
applicable), such as whether the LEO questioned the passenger, arrested 
the individual, or released the passenger. The SPOT referral database 
contains no personal identifying information about passengers. 



SPOT Has Been Deployed The SPOT program began with pilot tests in 2003 and 2004 at several New 
ill Phases England airports, in which TSA began using uniformed BDOs at airport 

checkpoints. After some initial pilot projects and test deployments, 644 



25 These requests would typically be made to the law enforcement entity employing the LEO, 
such as the airport authority police department. The department would have a computer 
that can access NCIC. According to the FBI's website, the NCIC database consists of 19 
files or databases. Seven are property files which contain records for articles, boats, guns, 
license plates, securities, vehicles, and vehicle and boat parts. Twelve are person files are 
the Convicted Sexual Offender Registry, Foreign Fugitive, Identity Theft, Immigration 
Violator, Missing Person, Protection Order, Supervised Release, Unidentified Person, U.S. 
Secret Service Protective, Violent Gang and Terrorist Organization, and Wanted Person 
Files. The Interstate Identification Index, which contains automated criminal history 
record information, is also accessible through the same network as NCIC. The Violent 
Gang and Terrorist Organization file includes the names of known or suspected terrorists. 

26 See 49 CFR §§ 1542.215, .217. 
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BDOs were deployed to 42 airports in the first phase of the program from 
November 2006 through June 2007. As of March 2010, about 3,000 BDOs 
utilizing SPOT were deployed at 161 of 457 TSA-regulated airports. 27 

BDO eligibility is restricted to TSOs with at least 12 months of TSO 
experience, or others with related security experience. Applicants must 
apply and be accepted into the BDO training program. The training 
includes 4 days of classroom courses, followed by 3 days of on-the-job 
training. BDOs must memorize all of the behaviors and appearances on 
the SPOT checklist, as well as the point value assigned to each, in order to 
be able to add these up to determine if a passenger should be sent to SPOT 
referral screening. BDO applicants must also pass a job knowledge test at 
the conclusion of the training. The test includes related multiple choice 
questions, true or false statements, and case-based scenarios. 



DHS Is Taking Action 
to Validate the 
Scientific Basis of 
TSA's SPOT Program 
but Opportunities 
Exist to Help Inform 
Future Program 
Decisions 



Although DHS is in the process of validating the way in which the SPOT 
program utilizes the science of behavior detection in an airport 
environment, TSA deployed SPOT nationwide before first determining 
whether there was a scientifically valid basis for using behavior and 
appearance indicators as a means for reliably identifying passengers as 
potential threats in airports. TSA reported that it deployed SPOT before a 
scientific validation of the program was completed in response to the need 
to address potential threats to the aviation system that would not 
necessarily be detected by existing layers of aviation security. TSA stated 
that no other large-scale U.S. or international screening program 
incorporating behavior- and appearance-based indicators has ever been 
rigorously scientifically validated. While TSA deployed SPOT on the basis 
of some risk-related factors, such as threat information and airport 
passenger volume, it did not use a comprehensive risk assessment to guide 
its strategy of selectively deploying SPOT to 161 of the nation's 457 TSA- 
regulated airports. TSA also expanded the SPOT program over the last 3 
years without the benefit of a cost-benefit analysis of SPOT. Additionally, 
TSA's strategic plan for SPOT could be improved by the inclusion of 
desirable characteristics identified in our prior work, such as risk 
assessment information, cost and resources analysis, and a means for 
collaboration with other key entities. 



27 TSA-regulated airports have regular commercial passenger service and comply with TSA 
regulations for passengers and their property in order to operate. 
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TSA Is in the Process of 
Validating the Scientific 
Basis Used to Identify 
Passengers with SPOT 
Behaviors 



TSA proceeded with deploying SPOT on a nationwide basis before 
determining whether the list of passenger behaviors and appearances 
underpinning the SPOT program were scientifically validated, and whether 
these techniques could be applied for counterterrorism purposes in an 
airport environment. In 2008, a report issued by the National Research 
Council of the National Academy of Sciences noted that behavior and 
appearances monitoring might be able to play a useful role in 
counterterrorism efforts but stated that a scientific consensus does not 
exist regarding whether any behavioral surveillance or physiological 
monitoring techniques are ready for use in the counterterrorist context 
given the present state of the science. 28 The report also stated that the 
scientific evidence for behavioral monitoring is preliminary in nature. 29 
According to the report, an information-based program, such as a behavior 
detection program, should first determine if a scientific foundation exists 
and use scientifically valid criteria to evaluate its effectiveness before 
going forward. The report added that programs should have a sound 
experimental basis and documentation on the program's effectiveness 
should be reviewed by an independent entity capable of evaluating the 
supporting scientific evidence. The report also stated that often scientists 
and other experts can help independently assess the scientific evidence on 
the effectiveness of a program. A contributor to the National Research 
Council report also stated that no conclusive research has been conducted 
to determine if behavior detection can be reliably used on a larger scale, 
such as in an airport setting, to identify persons intending to cause harm to 
the aviation system. 



While TSA and DHS's Science and Technology (S&T) Directorate officials 
agreed that SPOT was deployed before its scientific underpinnings were 
fully validated, they stated that no large-scale U.S. or international 
operational screening program incorporating behavior- and appearance- 



28 National Research Council, Protecting Individual Privacy in the Struggle Against 
Terrorists: A Framework for Assessment (Washington, D.C.: National Academies Press, 
2008). The report's preparation was overseen by the National Academy of Sciences 
Committee on Technical and Privacy Dimensions of Information for Terrorism Prevention 
and Other National Goals. Although the report addresses broader issues related to privacy 
and data mining, a senior National Research Council official stated that the committee 
included behavior detection as a focus because any behavior detection program could have 
privacy implications. 

^Specifically, the report states that the scientific support for linkages between behavioral 
and physiological markers and mental state is strongest for elementary states, such as 
simple emotions; weak for more complex states, such as deception; and nonexistent for 
highly complex states, such as when individuals hold terrorist intent and beliefs. 
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based indicators has been rigorously scientifically validated. These 
officials also questioned the findings of the National Research Council 
report and stated that the study lacked sufficient information for its 
conclusions because it did not consider recent findings from unpublished 
DHS, defense, and intelligence community studies. 30 However, National 
Research Council officials stated that an agency should be cautious about 
relying on the results of unpublished research that has not been peer 
reviewed, such as that generated by DHS and the defense and intelligence 
community, and using unpublished work as a basis for proceeding with a 
process, method, or program. 31 Moreover, we have previously reported 
that peer review is widely accepted as an important quality control 
mechanism that helps prevent the dissemination of potentially erroneous 
information. 32 

In addition to the unpublished research, TSA told us that the SPOT 
program was based on operational best practices from law enforcement, 
defense, and the intelligence communities. According to TSA officials, the 
agency based its choice of SPOT behavior, appearance, and deception 
indicators on existing research and training programs. For example, TSA 
cited research on emotions and their behavior indicators by Dr. Paul 
Ekman, 33 interviewing and interrogation by Stan Walters, 34 and nonverbal 



3 °DHS's S&T Directorate could not provide us with specific contacts related to the sources 
of this research. 

31 Peer review is the process of subjecting an author's scholarly work, research, or ideas to 
the scrutiny of others who are experts in the same field. Such review is considered a form 
of scientific validation. 

32 For example, we reported that the National Institutes of Health did not post its 
researchers' final reports because the risks associated with posting results that have not 
been scrutinized and validated by peer review are too great. See GAO, University 
Research: Most Federal Agencies Need to Better Protect against Financial Conflicts of 
Interest, GAO-04-31 (Washington, D.C.: November 2003). 

33 Dr. Ekman is professor emeritus of psychology at the University of California Medical 
School, San Francisco, and is considered one of the world's foremost experts on facial 
expressions. His books include: Emotions Revealed: Recognizing Faces and Feelings to 
Improve Communications and Emotional Life (New York: Holt and Company, 2003); 
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking the Face: A 
Guide to Recognizing Emotions from Facial Clues (Englewood Cliffs, N.J.: Prentice-Hall, 
1975). Dr. Ekman has published more than 100 articles. 

34 Mr. Walters is the author of the Principles of Kinesic Interview and Interrogation: 2 nd 
Edition as well as numerous training materials related to interviewing and interrogation 
techniques. 
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indicators by Dr. David Givens and Dr. Mark Frank as support for the 
choice of several of the behavior indicators. According to TSA, its 
development of the SPOT program was based on related DHS research 
and information from the training curricula of other federal agencies, such 
as the Federal Transit Administration and the Bureau of Alcohol, Tobacco, 
Firearms, and Explosives. 3 ' 

As with the SPOT behavior indicators, TSA told us that it sought input in 
creating the SPOT point scoring system from subject matter experts and 
from participants in TSA's SPOT working group, which consisted of law 
enforcement officials from agencies such as FBI, DEA, and local law 
enforcement officials. 38 While TSA officials said that they coordinated 
with relevant subject matter experts, such as Dr. Ekman, and based the 
SPOT scoring system on existing research and training programs, no 
validation of the behavior, appearance, and deception indicators was 
conducted prior to the deployment of SPOT in November 2006. According 
to TSA officials, they used professional judgment in developing the SPOT 



TJr. Givens is the director of the nonprofit Center for Nonverbal Studies, in Spokane, 
Washington. Dr. Givens is the author of Love Signals: A Practical Field Guide to the Body 
Language of Courtship (St. Martin's, New York, 2005) and Crime Signals: How to Spot a 
Criminal Before You Become a Victim (St. Martin's, 2008). The Center's Web site links to 
Dr. Givens' online reference tool, The Nonverbal Dictionary of Gestures, Signs and Body 
Language Cues. Dr. Givens said that he had did not know which nonverbal indicators had 
been selected by TSA for use in SPOT, that he had not been asked by TSA to review their 
choices from his list, or to review other aspects of the SPOT program. According to Dr. 
Givens, attempting to detect more than nine nonverbal indicators would be difficult for 
most individuals, even those trained in behavior detection. 

36 Dr. Frank is Associate Professor, Department of Communication, College of Arts and 
Sciences, at the University at Buffalo, State University of New York. He is on the Advisory 
Board of the University's Center for Unified Biometrics and Sensors, and has conducted 
research supported by DHS, the Defense Advanced Research Projects Agency, and the 
National Science Foundation. Dr. Frank told us that he had observed SPOT at an airport 
and had some coordination with TSA. However, he said that he had not reviewed the SPOT 
training curriculum or the SPOT scoring system. Dr. Frank stated that no study has been 
performed to validate use of behavior detection in an airport setting. 

3 According to DHS's S&T Directorate, it completed a study on suicide bomber indicators in 
July 2009. The program manager stated that they reviewed 157 documents and identified 
approximately 1,200 suicide indicators, which were similar to SPOT suicide bomber 
indicators. S&T stated that the study provides preliminary support for the detection of 
suicide bomber indicators and that SPOT represents best practices from defense and 
intelligence organizations. 

38 According to TSA, the FBI participated in discussions related to SPOT'S development in 
2006. 



Page 17 



GAO-10-763 Screening of Passengers by Observation Techniques 



point system and stated that the purpose of developing the scoring system 
was to increase the objectivity of the SPOT process. 

Dr. Ekman stated that, in his opinion, and after reviewing the scoring 
system and observing the program in operation, it was not clear whether 
the SPOT behaviors and appearances, and the related point system, could 
be used effectively in an airport environment because no credible 
validation research on this issue had been conducted. He noted, for 
example, that research is needed to identify how many BDOs are required 
to observe a given number of passengers moving at a given rate per day in 
an airport environment, or the length of time that such observation can be 
conducted before observation fatigue affects the effectiveness of the 
personnel. He commented that observation fatigue is a well-known 
phenomenon among workers whose work involves intense observation, 
and that it is essential to determine the duration of effective observation 
and to ensure consistency and reliability among the personnel carrying out 
the observations. 

DHS has recognized the need to conduct additional research to 
scientifically validate the use of the SPOT behavioral indicators in an 
airport environment. DHS's S&T Directorate began research in 2007 to 
determine if there is a statistically significant correlation between the 
SPOT behaviors exhibited by airport passengers and finding airport 
passengers with prohibited items (such as weapons), false documents, and 
illegal drugs or who pose a potential risk to aviation security. According 
to S&T, this research is expected to be completed in fiscal year 2011 and is 
to include three key elements. First, the study's purpose is to assess the 
reliability of the SPOT program by analyzing TSA's SPOT database to 
determine patterns of BDO scoring to measure consistency across BDOs, 
teams, locations, and other variables. Second, the study aims to compare 
the current implementation of SPOT to random passenger screening. 
Specifically, according to S&T officials, 130,000 passengers are to be 
randomly selected for additional SPOT referral screening. The study's 
design states that data collected about these passengers will be compared 
to data for passengers screened through the normal SPOT process. S&T 
officials expect that the results of this element of the study will provide a 
better understanding of how SPOT compares to random selection, as well 
as providing a baseline of each indicator present in the traveling public. 
Third, the study also aims to utilize live and video data, as available, to 
measure SPOT score ratings by BDOs of behaviors exhibited by 
passengers against ratings of the same passengers by subject matter 
experts. This element of the study could help determine whether BDOs 
are using, or are continuing to use, the SPOT score sheet correctly as time 
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passes after their initial training. According to S&T officials, the study is 
to form the basis for BDO performance and training requirements. 

The S&T Directorate reported some preliminary findings associated with 
this research in February 2008. The Directorate reported that although 
some of the existing literature supported the possibility of using 
behavioral and physiological cues, the results are not methodologically 
strong enough to support standardized applications in an operational 
setting. 39 The preliminary findings also noted that it is not known whether 
behavioral and physiological cues linked to deception in planning a hostile 
action will be the same or different as those indicators linked to deception 
by an individual after they have already engaged in a hostile action. 
However, an S&T program director stated that although early literature 
can be characterized as methodologically weak, more recent unpublished 
research sponsored by DHS, the Department of Defense, and the 
intelligence community is promising in that it has demonstrated some 
linkages between behavioral and physiological indicators and deception. 40 

In March 2009, the Under Secretary (Acting) for DHS's S&T Directorate 
testified that the Directorate had performed an initial validation of the 
behavior indicators used by BDOs. 41 The Under Secretary stated that this 
analysis provided statistically significant support that persons 
demonstrating select behavioral indicators are more likely to possess 
prohibited items and that behaviors can distinguish deceptive from 
nondeceptive individuals. According to S&T, this validation was the result 
of statistical analyses performed by S&T using operational data from the 
SPOT program database. However, we identified weaknesses in TSA's 
process for maintaining these data. For example, controls over the SPOT 
database to help ensure the completeness and accuracy of the data were 
missing. Specifically, the SPOT database did not have computerized edit 
checks built into the system to review the format, existence, and 
reasonableness of data. For example, we found that discrepancies existed 



American Institutes for Research, Behavioral Indicators Related to Deception in 
Individuals with Hostile Intentions: Interim Results (Washington, D.C.: February 2008). 
According to S&T officials, this review included research conducted prior to 2005. 

4 °DHS could not provide us with specific contacts related to the sources of this research; 
we were therefore unable to determine the extent to which it has demonstrated linkages 
between behavioral and physiological indicators and deception. 

"Statement of the Under Secretary (Acting), DHS S&T Directorate, before the 
Subcommittee on Homeland Security, Committee on Appropriations, U.S. House of 
Representatives, March 26, 2009. 
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between the number of passengers arrested by local law enforcement at 
the screening checkpoints and the number of screened passengers 
recorded as arrested. In another example, we found that the total number 
of LEO referrals differed from the number of passenger records with 
information on the reasons for LEO referral. Internal control standards 
state that controls should be installed at an application's interfaces with 
other systems to ensure that all inputs are received and are valid and that 
outputs are correct and properly distributed. 42 TSA officials explained 
these issues as data anomalies and planned to change instructions to staff 
entering data to reduce these problems. Although TSA is taking steps to 
update the SPOT database, which are discussed later in this report, the 
data used by S&T to conduct its preliminary validation of related 
behaviors lacked such controls. In addition, BDOs could not input all 
behaviors observed in the SPOT database because the database limits 
entry to eight behaviors, six signs of deception, and four types of 
prohibited items per passenger referred for additional screening. Because 
of these data-related issues, meaningful analyses could not be conducted 
to determine if there is an association between certain behaviors and the 
likelihood that a person displaying certain behaviors would be referred to 
a LEO or whether any behavior or combination of behaviors could be used 
to distinguish deceptive from nondeceptive individuals. As a result, TSA 
lacks assurance that the SPOT data can be used effectively to determine 
that the person poses a risk to aviation security. S&T has recognized 
weaknesses in the procedures for collecting data on passengers screened 
by SPOT and plans to more systematically collect data during its study by, 
for example, requiring BDOs to record more complete and accurate 
information related to a passenger referral immediately following 
resolution. 

The S&T study is an important step to determine whether SPOT is more 
effective at identifying passengers who may be threats to the aviation 
system than random screening. However, S&T's current research plan is 
not designed to fully validate whether behavior detection and appearances 
can be effectively used to reliably identify individuals in an airport 
terminal environment who pose a risk to the aviation system. For 
example, research on other issues, such as determining the number of 
individuals needed to observe a given number of passengers moving at a 
given rate per day in an airport environment or the duration that such 
observation can be conducted by BDOs before observation fatigue affects 



: GAO/AIMD-00-21.3.1. 
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effectiveness, could provide additional information on the extent to which 
SPOT can be effectively implemented in airports. In another example, Dr. 
Ekman told us that additional research could help determine the need for 
periodic refresher training since no research has yet determined whether 
behavior detection is easily forgotten or can be potentially degraded with 
time or lack of use. While S&T officials agree on the need to validate the 
science of behavior detection programs, they told us that some of these 
other issues could be examined in the future but are not part of the 
current plan due to time and budgetary constraints. According to S&T, 
some additional analysis is underway to inform the current BDO selection 
process. This analysis is intended to provide information on the 
knowledge, skills, abilities, and other characteristics of successful BDOs. 
Since the analysis is scheduled for completion in May 2010, it remains 
unclear to what extent the findings will help to validate the science related 
to SPOT. While we recognize the potential benefits of these efforts, we 
believe that an assessment by an independent panel of experts of the 
planned methodology of DHS's study could help DHS assess the costs and 
benefits associated with a more comprehensive methodology designed to 
fully validate the science related to SPOT. Our prior work has 
recommended the use of such independent panels for comprehensive, 
objective reviews of complex issues. 43 In addition, according to the 
National Research Council, an independent panel could provide an 
objective assessment of the methodology and findings of DHS's study to 
better ensure that SPOT is based on validated science. Thus, an 
independent panel of experts could help DHS develop a comprehensive 
methodology to determine if the SPOT program is based on valid scientific 
principles that can be effectively applied in an airport environment for 
counterterrorism purposes. 



See GAO, Oil and Gas Royalties: The Federal System for Collecting Oil and Gas 
Revenues Needs Comprehensive Reassessment, GAO-08-691 (Washington, D.C.: Sept. 3, 
2008). GAO, Combating Nuclear Smuggling: Additional Actions Needed to Ensure 
Adequate Testing of Next Generation Radiation Detection Equipment, GAO-07-124T 
(Washington, D.C.: Sept. 18, 2007). GAO, Space Operations: NASA Efforts to Develop and 
Deploy Advanced Spacecraft Computers, GAO/IMTEC-89-17 (Washington, D.C.: Mar. 31, 
1989). GAO, Quadrennial Defense Review: Future Reviews Could Benefit from Improved 
Department of Defense Analyses and Changes to Legislative Requirements, GAO-07-709 
(Washington, D.C.: Sept. 14, 2007). GAO, Coast Guard: Challenges for Addressing Budget 
Constraints, GAO/RCED-97-110 (Washington, D.C.: May 1997). 
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SPOT Was Deployed 
Nationwide on Basis of 
Threat, but Without a 
Comprehensive Risk 
Assessment 



According to DHS's National Infrastructure Protection Plan (NIPP), risk 
assessments are to be documented, reproducible (so that others can verify 
the results), defensible (technically sound and free of significant errors), 
and complete. The NIPP states that comprehensive risk assessments are 
necessary for determining which assets or systems face the highest risk, 
for prioritizing risk mitigation efforts and the allocation of resources, and 
for effectively measuring how security programs reduce risks. For a risk 
assessment to be considered complete, the NIPP states that it must 
specifically assess threat, vulnerability, and consequence; 44 after these 
three components have been assessed, they are to be combined to 
produce a risk estimate. 45 



According to TSA, SPOT was deployed to TSA-regulated airports on the 
basis of threat information in TSA's Current Airport Threat Assessment 
list. 46 TSA deployed SPOT to 161 of 457 TSA-regulated airports. TSA 
officials told us that this selective deployment creates unpredictability for 
persons seeking to cause harm to the aviation system because they would 
not know which airports had BDO teams and because BDOs are 
occasionally sent out to the smaller airports that do not have BDOs on a 
permanent basis. Although TSA's selective deployment of SPOT was 
based on threat information, TSA did not conduct vulnerability and 
consequence assessments to inform the deployment of BDOs. As a result, 



"DHS's NIPP defines risk as a function of threat, vulnerability, and consequence. Threat is 
an indication of the likelihood that a specific type of attack will be initiated against a 
specific target or class of targets. Vulnerability is the probability that a particular attempted 
attack will succeed against a particular target or class of targets. Consequence is the effect 
of a successful attack. 

45 As updated in 2009, the NIPP states that to be complete a risk assessment is to assess 
threat, vulnerability, and consequence for every defined risk scenario. However, because 
the original 2006 version of the NIPP described risk assessments that included all three 
components as "credible," our previous reports use this term rather than "complete" (see 
GAO, Transportation Security: Comprehensive Risk Assessments and Stronger Internal 
Controls Needed to Help Inform TSA Resource Allocation, GAO-09-492 (Washington, D.C.: 
Mar. 27, 2009)). 

4t We reported in March 2009 that TSA does not assign uncertainty or varying levels of 
confidence associated with the intelligence information used to identify threats to the 
transportation sector and guide TSA's planning and investment decisions. Since TSA's 
intelligence products have not assigned confidence levels to its analytic judgments, it is 
difficult for TSA to correctly prioritize its tactics and investments based on uncertain 
intelligence. In March 2009, we recommended that TSA work with the Director of National 
Intelligence to determine the best approach for assigning uncertainty or confidence levels 
to analytic intelligence products and to apply this approach. TSA agreed with this 
recommendation and said it has begun taking action to address it. See GAO-09-492. 
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it could not combine the results to conduct a comprehensive risk 
assessment to inform the deployment of BDOs to those airports with the 
highest risks. 

TSA officials told us that while they have not completed a comprehensive 
risk assessment for airport security, they have prepared and are currently 
reviewing a draft of a comprehensive, scenario-based Aviation Modal Risk 
Assessment — known as the AMRA — which is to serve as a comprehensive 
risk assessment for aviation security. 4 ' According to TSA officials, the 
AMRA is to address all three elements of risk for domestic commercial 
aviation, general aviation, and air cargo. 48 Although TSA planned to 
release the AMRA in February 2008, it now expects to finalize the AMRA in 
2010. According to TSA, the AMRA may help provide information for the 
prioritization of BDO deployment within airports, but could not provide 
specifics on how it would do so. Further, TSA officials noted that 
information from AMRA would inform BDO deployment in conjunction 
with other TSA priorities not related to SPOT. 49 Since the AMRA is not yet 
complete, it is not clear whether it will provide the risk analysis — 
including assessments of vulnerability and consequence — needed to 
inform TSA's decisions and planning for any revisions or future 
deployment of SPOT. If AMRA lacks information relevant to the 
deployment of SPOT and further research determines that SPOT has a 
scientifically validated basis for using behavior detection for 
counterterrorism purposes in the airport environment, then conducting a 
comprehensive risk assessment of airports could strengthen TSA's ability 
to establish priorities and make cost-effective resource decisions 



The AMRA is being developed by TSA pursuant to the National Strategy for Aviation 
Security, which was issued according to Homeland Security Presidential Directive-16. 
HSPD-16 directs the production of an overarching national strategy to optimize and 
integrate government-wide aviation security efforts. AMRA was previously known as the 
Air Domain Risk Assessment or ADRA. 

48 Commercial aviation includes that sector of the nation's civil aviation system that 
provides for the transportation of individuals by scheduled or chartered operations for a 
fee, including air carriers and airports. General aviation includes all civil aviation other 
than commercial and military operations, including flight operations such as 
personal/family transportation, emergency services, wildlife and land surveys, traffic 
reporting, agricultural aviation, firefighting, and law enforcement. Air cargo is defined as 
cargo carried on passenger and all-cargo aircraft. 

4 °In addition, TSA states that its risk management analysis toolset may also be useful in 
prioritizing BDO deployments since the toolset analyzes various scenarios for which the 
use of BDOs may be applicable. 
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regarding the deployment of BDOs to those airports deemed to have the 
highest priority risks. 



TSA Deployed SPOT 
Nationwide Without 
Conducting a Cost-Benefit 
Analysis but Such an 
Analysis Could Help 
Inform Program Decisions 
Moving Forward 



DHS and other federal guidance recommend conducting a cost-benefit 
analysis before implementing new programs to avoid unnecessary costs 
and identify the best way to achieve goals at the lowest costs among 
potential alternatives. Our prior work has also supported the use of cost- 
benefit analyses during retrospective reviews to validate the agency's 
original assumptions regarding costs and benefits. 50 In addition, the DHS 
February 2006 Cost-Benefit Analysis Guidebook and OMB guidance both 
recommend the use of cost-benefit analysis, both in the planning stage for 
a program, and when significant milestones or financial options are to be 
assessed. 51 The DHS Guidebook states that a cost-benefit analysis is 
designed to identify optimal financial solutions among competing 
alternatives. OMB guidance also identifies cost-benefit analysis as one of 
the key principles to be considered when making capital expenditures, 
that expected benefits of proposed actions should be explained, and that a 
baseline should be identified discussing costs and benefits in comparison 
with clearly defined alternatives. DHS's 2006 and 2009 NIPPs also state 
that priority is to be given to those protective measures that provide the 
greatest mitigation of risk for the resources that are available. The DHS 
NIPPs add that effective protective programs seek to use resources 
efficiently by focusing on actions that offer the greatest mitigation of risk 
for any given expenditure. In addition, measuring cost effectiveness of 
SPOT was a key TSA goal in an October 2005 version of the SPOT strategic 
plan. 



Although the DHS and OMB guidance recommend that a cost-benefit 
analysis be conducted prior to deploying a program nationwide — and 
potentially incurring substantial costs — TSA did not conduct such an 
analysis of SPOT to inform its pilot testing prior to full-scale nationwide 
deployment. In early 2003, TSA began conducting a pilot test of the SPOT 
program at Boston Logan airport to better understand the benefits of the 
program. According to Boston Logan's Federal Security Director, the 



" See GAO, Reexamining Regulations: Opportunities Exist to Improve Effectiveness and 
Transparency of Retrospective Reviews, GAO-07-791 (Washington, D.C.: July 16, 2007). 

Bl DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February 2006); OMB, Circular 
No. A-94, Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs 
(Washington, D.C.: October 1992); OMB, Circular No. A-4, Regulatory Analysis 
(Washington, D.C.: September 2003). 
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primary purpose of this pilot test was to understand the potential of the 
program, not to validate its success. 52 TSA officials stated that the 
program had several benefits, one of which was its "negligible cost." 
However, TSA did not analyze the pilot test results to determine if SPOT 
was more cost effective than other alternatives, such as random screening 
of passengers. In October 2004, TSA implemented additional pilot 
programs in Providence, Rhode Island and Portland, Maine with the goal 
of providing Federal Security Directors with an additional layer of security 
to identify high-risk passengers for additional screening using behavior 
detection techniques. TSA concluded that the pilot program was 
successful and cited several security benefits of these pilots. For example, 
TSA personnel in Providence identified two individuals in possession of 
illegal drugs, who were then arrested. Law enforcement also arrested 
another individual referred to them for providing a fraudulent passport. In 
another example, BDOs in Portland discovered a passenger with multiple 
passports and a hidden luggage compartment. The passenger was 
interviewed by LEOs and later released. 

TSA determined that these initial pilot tests at three airports were 
successful without comparing pilot test data to other possible security 
alternatives. For example, the results of random screening of passengers 
at the pilot airports could have provided TSA with objective baseline data. 
Specifically, these data could have been compared to data collected during 
the SPOT pilots to determine if SPOT was more effective than random 
screening in detecting passengers who pose a potential risk to aviation 
security. TSA concluded that the pilot tests were successful because pilot 
airports were able to easily incorporate SPOT into their security program, 
train personnel in SPOT, and implement procedures for an additional layer 
of security according to TSA. 

TSA conducted additional pilot tests at the Minneapolis-St. Paul, 
Minnesota and Bangor, Maine airports in October 2005. TSA also deployed 
the program to nine additional airports in response to TSA's holiday 
preparedness plan in December 2005 to further operationally test the 
program. Senior SPOT program officials explained that TSA did not 
conduct an analysis of the pilot testing because the program was in its 



A pilot test is a preliminary test or study to try out procedures and discover problems 
before the main study begins. This enables researchers to make last minute corrections and 
adjustments. In a pilot, the entire study with all of its instruments and procedures is 
conducted in miniature. See W.P. Vogt, Dictionary of Statistics and Methodology: A 
Nontechnical Guide for the Social Sciences (Newbury Park: Sage Publications, 1993). 
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infancy and officials were focused on deploying SPOT to additional 
airports. Since that time, TSA has not conducted a cost-benefit analysis, 
which could help the agency establish the value of the program relative to 
other layers of aviation security. Moreover, a cost-benefit analysis could 
also be useful considering recent program growth. For example, from 
fiscal year 2007 through fiscal year 2009, TSA allotted about $383 million 
for SPOT. During this period, SPOT's share of TSA's total screening 
operations budget increased from 1 percent to 5 percent. 53 The conference 
report accompanying the fiscal year 2010 DHS appropriations act 
designates $212 million of the appropriated aviation security funding for 
the SPOT program. 54 A cost-benefit analysis could have provided TSA 
management with analysis on whether this allocation was a prudent 
investment, as well as whether this level of investment in SPOT is 
appropriate. Figure 3 shows the growth in the budget and personnel 
numbers for SPOT from fiscal years 2007 through 2010. 



°' 5 The increase rate for TSA's other screening operations combined was about 0.27 percent 
from fiscal year 2007 to fiscal year 2009 (from $3,727 billion to $3,737 billion, a $10 million 
increase). The screening operations account includes privatized screening; passenger and 
baggage screener performance, compensation, and benefits; screener training and other; 
human resource services; and checkpoint support. 

"See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.). 
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Figure 3: Budget and Personnel Growth in the SPOT Program, Fiscal Years 2007 
through 2010 



Dollars In millions 
250 



200 



$212 



$198 



150 



100 



$144 



50 $41 



BDO allocations 
3,500 

3,000 

2,500 

2,000 



1,500 



1,000 



500 



2007 2008 2009 2010 

Fiscal years 



2,860 



2,986 



2,011 



589 



2007 2008 2009 2010 

Fiscal years 



| | Total actual cost 

| Total appropriated 

Source: GAO analysis of TSA data. 



Actual BDO allocation 
Total appropriated 



Note: The actual BDO allocation for fiscal year 2009 is as of June 2009. The appropriated amount for 
SPOT for fiscal year 201 is the amount reflected in the conference report accompanying the fiscal 
year 2010 DHS appropriations act. The appropriated amounts prior to fiscal year 2010 cannot be 
determined because funding was appropriated as a lump sum with funding for other screeners and 
the relevant conference reports did not allocate a specific amount for SPOT. BDO allocation figures 
are full-time equivalents. 
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SPOT'S Strategic Plan 
Could be Strengthened by 
Addressing Key 
Characteristics of a 
Successful Strategy 



Our previous work, 55 and the Government Performance and Results Act, 56 
set forth several key elements of a strategic plan. Such plans can guide 
agencies in planning and implementing an effective government program. 
Table 1 summarizes the desirable characteristics of an effective strategic 
plan, as identified in our prior work. In April 2009, we reported that these 
characteristics are the starting point for developing a strategic plan. 57 



Table 1 : Summary of Desirable Characteristics for Developing a Strategic Plan 



Desirable characteristic 


Description 


Purpose, scope, and methodology 


Addresses why the plan was produced, the scope of its coverage, 




and the process by which it was developed. 


Problem definition and risk assessment 


Addresses the particular problems and threats the plan is directed 




towards. 



Goals, subordinate objectives, activities, and performance 
measures 



Addresses what the plan is trying to achieve, steps to achieve 
those results, as well as the priorities, milestones, and 
performance measures to gauge results. 



Resources, investments, and risk management 



Addresses what the plan will cost, the sources and types of 
resources and investments needed, and where resources and 
investments should be targeted based on balancing risk 
reductions with cost. 



Organizational roles, responsibilities, and coordination 



Addresses who will implement the plan, what their roles will be 
compared to others, and mechanisms for them to coordinate their 
efforts. 



Integration and implementation 



Addresses how the plan relates to the agency's other goals, 
objectives, and activities, to other federal and nonfederal entities 
involved in implementation or coordination, and their plans to 
implement the strategic plan. 



Source: GAO analysis based on GAO-09-369 and GAO-04-408T. 



TSA officials at Boston Logan airport told us that they completed the first 
strategic plan for SPOT in 2006. The strategic plan was last updated in 
March 2007. The March 2007 plan includes some of the desirable 
characteristics described above, such as an overall purpose. However, 
incorporating additional characteristics of an effective strategic plan could 
enhance the plan's usefulness in program management and resource 
allocation decisions to effectively manage the deployment of SPOT if TSA 



s5 GAO-04-408T. 

56 Pub. L. No. 103-62, 107 Stat. 285 (1993). 

"See GAO, National Preparedness: FEMA Has Made Progress, but Needs to Complete and 
Integrate Planning, Exercise, and Assessment Efforts, GAO-09-369 (Washington, D.C.: 
Apr. 30, 2009). 
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determines that the program has a scientifically valid basis. TSA officials 
stated that they believed the plan was sufficiently comprehensive to 
develop a national program, such as SPOT. However, these officials told 
us that the plan was not updated after TSA expanded the program in 2008 
and 2009. They also stated that the program's focus remained on 
deploying SPOT to additional airports. Our assessment of the extent to 
which the SPOT strategic plan addresses these characteristics is presented 
below. 

Purpose, scope, and methodology: The SPOT strategic plan addresses 
why the plan was developed (i.e., purpose) and the scope of its coverage. 
Specifically, the plan describes a strategy to utilize behavior detection 
screening as an additional layer of security. The plan also notes that the 
primary focus is to expand SPOT in the aviation environment while also 
developing a capability to deploy BDOs to support security efforts in all 
modes of transportation. However, the plan does not discuss the process 
by which it was developed (i.e., methodology). According to TSA, officials 
responsible for developing the plan received input from relevant 
stakeholders at Boston Logan airport and TSA headquarters. We believe 
incorporating the methodology into the plan could make the document 
more useful to TSA and other organizations, such as local law 
enforcement, responsible for implementing the plan. 

Problem definition and risk assessment: The plan addresses the 
particular threat it is directed towards. Specifically, the plan describes the 
need to implement SPOT to counter terrorist activities, improve security, 
and incorporate additional layers of protection within aviation security. 
However, the plan does not incorporate risk assessment information to 
identify priorities or guide program implementation because TSA has not 
conducted a comprehensive risk assessment related to the deployment of 
SPOT. 58 Using available risk assessment information to inform the 
development of a strategic plan would help ensure that clear priorities are 
established and focused on the areas of greatest need. Specifically, 
incorporating the results of a risk assessment in the program's strategic 
plan could help inform TSA's decisions such as whether to deploy SPOT to 
additional TSA-regulated airports, to shift SPOT teams from one airport to 
another, or to remove SPOT at airports where the benefit of addressing the 
risk does not outweigh the costs, as well as to identify and communicate 



a8 TSA, Strategic Plan for Behavior Detection Program (Washington, D.C.: 2007). 
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the risks to aviation security if SPOT was not deployed to all TSA- 
regulated airports. 

Goals, subordinate objectives, activities, and performance 
measures: The plan outlines several goals, objectives, and activities for 
the SPOT program to achieve. For example, the plan outlines a goal to 
develop multimodal partnerships, including at the local level, to support 
SPOT. An associated objective for this goal includes identifying and 
fostering advocates within each mode of transportation by developing 
transportation, intelligence, and law enforcement working groups with 
relevant officials to share information and foster cooperation. The plan 
also includes a goal to develop and implement performance measures for 
SPOT. However, the plan did not include performance measures for 
SPOT. Incorporating performance measures into the plan could help TSA 
officials measure progress in implementing the plan's goals, objectives, 
and activities. 

Resources, investments, and risk management: The plan does not 
identify the costs and resources needed to achieve program objectives 
discussed in the plan. Incorporating information about cost and resources 
would facilitate TSA's ability to allocate resources across programs 
according to priorities and constraints, track costs and performance, and 
shift such investments and resources as appropriate. 

Organizational roles, responsibilities, and coordination: The SPOT 
program relies on a close partnership with law enforcement officers at 
airports. TSA provides briefings to law enforcement on the SPOT 
program, and TSA officials conduct outreach efforts to local law 
enforcement as needed. The SPOT SOP guidance and SPOT training 
include guidance about ensuring that LEOs receive complete and accurate 
information about each SPOT referral. However, while the strategic plan 
identifies TSA officials and offices as responsible parties for implementing 
the strategic plan, it does not provide guidance on how to effectively link 
the roles, responsibilities, and capabilities of federal, state, and local 
officials providing program support. Moreover, although SPOT SOP 
guidance discusses the need for BDOs to coordinate with other TSA 
personnel, such as TSOs and TDCs, TSA does not identify their roles and 
responsibilities in regards to the SPOT program in the program's strategic 
plan. Integrating these elements into the strategic plan could help to 
clarify the relationships between these various implementing parties, 
which would thereby increase accountability and improve the 
effectiveness of implementation. 
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Integration and implementation: The SPOT strategic plan does not 
discuss how its scope complements, expands upon, or overlaps with other 
related strategic documents. For example, TSA's April 2008 Office of 
Security Operations Organizational Business Plan for Fiscal Year 2010 
describes how its goals — including those for SPOT — relate to DHS and 
TSA strategic goals. 59 However, TSA does not link goals in the SPOT 
strategic plan with other related strategic documents, such as the Aviation 
Implementation Plan of DHS's Transportation Systems Sector-Specific 
Plan 60 and the Passenger Checkpoint Screening Program Strategic Plan. 61 
By linking goals in its SPOT strategic plan to other TSA efforts, TSA could 
better ensure that the program's objectives are integrated with other TSA 
security programs and that resources are used effectively by minimizing 
any unnecessary duplication with these other actions. 



More Fully and 
Consistently Utilizing 
Available Information 
Technology Could 
Enhance TSA's Ability 
to Identify Threats to 
the Aviation System 



Inconsistencies in the use of available information technology to aid in the 
collection and recording of data on passengers by BDOs during referrals to 
LEOs, lack of guidance on, or a mechanism for, BDOs to request the TSA's 
Transportation Security Operations Center to run the names of passengers 
exhibiting suspicious behaviors against law enforcement and intelligence 
databases, and the Center's not checking all of the databases available to 
it — have limited TSA's ability to identify potential terrorist threats to the 
aviation system. 62 Among other information, these databases include 
terrorism-related watch lists. 



° 9 TSA, Office of Security Operations, Strategy Deployment Organizational Business Plan 
for Fiscal Year 2010 (Washington, D.C.: Apr. 8, 2008). 

"Within the Transportation System Sector-Specific Plan, the aviation implementation plan 
outlines transportation security goals and key objectives with associated programs within 
the aviation community. The plan notes that SPOT is intended to identify suspicious 
activities within the aviation domain. 

01 TSA issued its Passenger Checkpoint Screening Program Strategic Plan in August 2008 to 
outline its strategy and approach for implementing advanced security capabilities in airport 
checkpoints using a combination of people, processes, and technology at all airport 
checkpoints. The plan cites TSA's behavior detection capability as one of three strategic 
initiatives. 

62 The Transportation Security Operations Center is the central operations and information- 
gathering point for TSA across the nation; it serves as a 24/7-point of contact for all 
transportation security concerns, providing access to multiple criminal justice and 
intelligence-related databases. 
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Systematic Collection of 
Data on Passengers 
Identified Through the 
SPOT Program Could be 
Improved to Better 
Identify Activity 
Potentially Harmful to the 
Aviation System 



TSA is not fully utilizing the resources it has available to systematically 
collect the information obtained by BDOs on passengers whose behaviors 
and appearances resulted in either SPOT referral screening, or in a referral 
to LEOs, and who thus may pose a risk to the aviation system. TSA's July 
2008 Privacy Impact Assessment on the TSA Transportation Security 
Operations Center, and its August 2008 Privacy Impact Assessment on 
SPOT, state that information may be obtained by BDOs to check an 
individual's identity against intelligence, terrorist, and law enforcement 
databases and to permit intelligence analysts to conduct trend analysis. 63 

The August 2008 SPOT Privacy Impact Assessment states that information 
about a passenger who has exceeded the SPOT behavior threshold, 
leading to LEO referral, may be collected and entered into DHS's 
Transportation Information Sharing System. 64 According to the SPOT 
Privacy Impact Assessment, information collected may be submitted to the 
Transportation Information Sharing System database for analysis, and, 
through it to other linked intelligence databases and the intelligence 
analysts who study them, to detect, deter, and defeat a criminal or terrorist 
act in the transportation domain before it occurs. The SPOT Privacy 
Impact Assessment notes that terrorist acts that threaten transportation 
security are most vulnerable in the planning stages and that the timely 
passage of SPOT referral information may assist in identifying such efforts 
before they become operational. A June 2008 Transportation Information 
Sharing System Privacy Impact Assessment similarly states that one goal is 
to use the system data to find trends and patterns that may indicate 
preoperational terrorist or criminal activity — that is, to "connect the dots" 
about a planned terrorist attack or criminal enterprise. Information in 
TSA's Transportation Information Sharing System is primarily activity or 
behavioral information but may also contain personal information 
regarding the individuals identified by the BDO through SPOT. According 



K5 DHS, Privacy Impact Assessment for the TSA Operations Center Incident Management 
System (Washington, D.C.: July 8, 2008), and Privacy Impact Assessment for the Screening 
of Passengers by Observation Techniques (SPOT) Program (Washington, D.C.: Aug. 5, 
2008). 

M DHS, Privacy Impact Assessment for the Transportation Information Sharing System 
(Washington, D.C.: June 2008). The Transportation Information Sharing System is a 
database owned by the TSA's FAMS component; the data entered into it may be shared 
with other federal, state, or local law enforcement and law enforcement support entities. 
Federal air marshals file reports related to the observation of suspicious activities and 
input this information, as well as incident reports submitted by airline employees and other 
individuals within the aviation domain, into the Transportation Information Sharing 
System. 
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to TSA, BDOs do not analyze the data obtained during referrals; if they 
have the appropriate training, they may enter the data by computer into 
the Transportation Information Sharing System, where they can be 
analyzed by intelligence analysts. Other appropriately trained and 
officially designated TSA officials, such as Federal Security Directors, may 
also enter data into the system. 

According to TSA, a 2008 pilot program it conducted that involved BDOs 
entering data into the Transportation Information Sharing System 
database was "invaluable," in part because over 40 referrals have since 
been passed on to other LEO organizations for further investigation, most 
of which came from BDO input. A February 2006 TSA memorandum 
describes the Transportation Information Sharing System as "a critical 
element in the success of SPOT" because it provides the necessary 
platform for the reporting of information obtained as a result of SPOT 
referrals. TSA noted that through the use of the Transportation 
Information Sharing System, two different BDO teams had separately 
identified and selected the "same extremist" for secondary questioning. 65 
TSA officials also told us about an incident in which an individual sought 
to board an aircraft with a handgun on two separate occasions, at two 
different airports. Although the handgun was detected both times, the 
individual was released after providing what seemed to be a credible 
explanation. After the second incident, however, intelligence analysts 
who reviewed the system information saw that this individual had tried 
twice in 2 weeks to bring a weapon onto an aircraft. A LEO was 
dispatched to the person's home, and an arrest was made. Without the 
data inputted into the system both times, no pattern would have been 
detected by the analysts, according to TSA. Although the pilot program 
illustrated the benefits of BDOs entering data into the system, access to 
the system was not expanded to all SPOT airports in 2008 or 2009. 

Internal control standards call for management to develop policies, 
procedures, and techniques to help enforce management directives. TSA 
does not provide official guidance on how or when BDOs or other TSA 
personnel should enter data into the Transportation Information Sharing 
System or which data should be entered. Official guidance on what data 
should be entered into the system on passengers could better position TSA 



65 Because the SPOT program has not been scientifically validated, it cannot be determined 
if these anecdotal results were better than if passengers had been pulled aside at random, 
rather than as a consequence of being identified for further screening by BDOs. 



Page 33 



GAO-10-763 Screening of Passengers by Observation Techniques 



personnel to be able to consistently collect information to facilitate 
synthesis and analysis in "connecting the dots" with regard to persons who 
may pose a threat to the aviation system. 

On March 18, 2010, TSA officials told us that TSA recognizes the value of 
recording SPOT incidents for the purposes of intelligence gathering. As a 
result, TSA decided that certain data would be entered into the 
Transportation Information Sharing System, and would, in turn, be 
analyzed as a way to potentially "connect the dots" with other 
transportation security incidents. 66 

TSA officials said that the Federal Security Director at each SPOT airport 
has been given the discretion to decide which personnel should have 
access to the Transportation Information Sharing System. However, TSA 
has not developed a plan detailing how many personnel would have access 
to the system, or when they would have access at SPOT airports. TSA 
officials said that training is currently being provided to personnel 
responsible for using the system at all SPOT airports although they did not 
provide information on the number being trained. 

Standard practices for defining, designing, and executing programs 
include developing a road map, or program plan, to establish an order for 
executing specific projects needed to obtain defined programmatic results 
within a specified time frame. However, TSA stated that it has not 
developed a schedule or milestones by which database access will be 
deployed to SPOT airports, or a date by which access at all SPOT airports 
will be completed. Setting milestones for expanding Transportation 
Information Sharing System access to all SPOT airports, and setting a date 
by which the expansion will be completed, could better position TSA to 
identify threats to the aviation system that may otherwise go undetected 
and help TSA track its progress in expanding Transportation Information 
Sharing System access as management intended. 



M Some details about the process were deleted because TSA considered them to be 
Sensitive Security Information. 



Page 34 



GAO-10-763 Screening of Passengers by Observation Techniques 



Internal control standards state that policies, procedures, techniques, and 
other mechanisms are essential to help ensure that actions are taken to 
address program risks. 6 ' The current process makes the BDOs dependent 
on the LEOs with regard to the timeliness that LEOs respond to BDO calls 
for service, as well as with regard to whether the LEOs choose to question 
the passengers referred to them or conduct a background check. Our 
analysis of the SPOT referral database found a wide variation in the 
percent of times that LEOs responded to calls for service at SPOT 
airports. 68 Moreover, if a local LEO decides to run a background check on 
a passenger referred to them, they would be accessing the FBI's NCIC and 
not other intelligence and law enforcement databases. 

Although LEOs may not always respond to calls for service, question 
passengers, or check passenger names against databases available to TSA, 
TSA has not developed a mechanism allowing BDOs to send information 
to the Transportation Security Operations Center about passengers whose 
behavior indicates that they may be a possible threat to aviation security. 
According to TSA's July 2008 Transportation Security Operations Center 
Privacy Impact Assessment, passenger information may be submitted to 
the Transportation Security Operations Center to ascertain, as quickly as 
possible, the individual's identity, whether they are already the subject of a 
terrorist or criminal investigation, or to analyze suspicious behavior that 
may signal some form of preoperational surveillance or activity. 69 

Our survey of Federal Security Directors at SPOT airports found a notable 
inconsistency in the rates at which BDOs at different airports contacted 
the Transportation Security Operations Center. 70 Developing additional 
guidance in the SPOT operating procedures could help improve 
consistency in the extent to which BDOs utilize Transportation Security 
Operations Center resources. Given the range of responses we received 
from SPOT airports about whether the BDOs contact the Transportation 
Security Operations Center to verify passenger identities and run their 



Guidance on and a 
Mechanism for Running 
Names of Referred 
Passengers Through the 
Databases Available to the 
Transportation Security 
Operations Center Could 
Help Improve SPOT 
Practices 



67 GAO/AIMD-00-21.3.1. 

0S Some details from our analysis were deleted because TSA considered them to be Sensitive 
Security Information. 

ra This information can be submitted about individuals whose suspicious activity resulted in 
BDO or LEO referral. See TSA's July 2008 Transportation Security Operations Center 
Privacy Impact Assessment. 

™Some details of our survey results were deleted because TSA considered them to be 
Sensitive Security Information. 
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names against terrorist and intelligence databases and the inconsistencies 
identified related to LEO responses to BDO requests for service, 
developing a standard mechanism and providing BDOs with additional 
guidance could help TSA achieve greater consistency in the SPOT process. 
Such a mechanism would provide designated TSA officials with a means of 
verifying passenger identities and help them determine whether a 
passenger was the subject of a terrorist or criminal investigation and thus 
posed a risk to aviation security. 

Standards for internal control state that effectively using available 
resources, including key information databases, is one element of 
functioning internal controls. 71 In this connection, it is widely recognized 
among intelligence entities and police forces that a capability to "run" 
names against databases that contain criminal and other records is a 
potentially powerful tool to both identify those with outstanding warrants 
and to help discover an ongoing criminal or security-related incident. 
Additionally, TSA recommended in an April 2008 Organizational Business 
plan for its Office of Security Operations that the SPOT program should 
establish a mechanism and policy for allowing real-time checks of federal 
records for individuals whose behavior indicates they may be a threat to 
security. 72 The Office of Security Operations plan also states that BDOs 
should communicate the data to U.S. intelligence centers, with the 
purpose of permitting rapid communication of this information to local 
LEOs to take action. However, TSA officials told us that because of safety 
concerns, the Transportation Security Operations Center does not provide 
information from database checks directly to BDOs because BDOs are not 
LEOs, are unarmed, and do not have the training needed to deal with 
potentially violent persons. 73 If the mechanism discussed in the Office of 



71 See GAO/AIMD-00-21.3.1. For example, information should be recorded and 
communicated to management and others within the entity who need it and in a form and 
within a time frame that enables them to carry out their internal control and other 
responsibilities. Further, effective information technology management is critical to 
achieving useful, reliable, and continuous recording and communication of information. 

72 TSA, Strategy Deployment, Organizational Business Plan, Office of Security 
Operations, Fiscal Year 2010 (Washington, D.C.: April 2008). According to TSA, the Office 
of Security Operations is the operational arm of TSA and employs the largest TSA 
workforce. It is responsible for airport checkpoint and baggage screening operations as 
well as other special programs designed to secure all assigned transportation modes. 

73 In March 2010, TSA told us that over the next 18 months, it will expand access to 
information classified up to the "Secret" level to an additional 10,000 TSA personnel, 
including all BDOs, all SPOT Transportation Security Managers (who are responsible for 
the local operations of the SPOT program and supervision of the BDOs), and all 
Supervisory TSOs (who directly supervise TSOs and the screening process). 
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Security Operations business plan were implemented, it would allow the 
Transportation Security Operations Center to use BDO information to 
conduct real-time record checks of passengers and communicate the 
results to LEOs for action. Such a mechanism could increase the chances 
to detect ongoing criminal or terror plans. 



TSA's Transportation 
Security Operations Center 
Does Not Use All Database 
Resources When 
Contacted 



The final report of the National Commission on Terrorist Attacks Upon the 
United States (the "9/11 Commission Report") recommends that in 
carrying out its goal of protecting aviation, TSA should utilize the larger 
set of information maintained by the federal government, that is, the entire 
Terrorist Screening Database — the U.S. government's consolidated watch 
list that contains information on known or suspected international and 
domestic terrorists — as well as other government databases, such as 
intelligence or law enforcement databases. 74 However, the Transportation 
Security Operations Center is not using all the resources at its disposal to 
support BDOs in verifying potential risks to the aviation system. This 
reduces the opportunities to "connect the dots" that would increase the 
chances of detecting terrorist attacks in their planning stage, which the 
SPOT Privacy Impact Assessment states is when they are the most 
vulnerable. 



According to TSA, the Transportation Security Operations Center has 
access to multiple law enforcement and intelligence databases that can be 
used to verify the identity of airline passengers; these include among 
others: 75 



1. the Selectee list, which identifies persons who must undergo enhanced 
screening at the checkpoint prior to boarding; 

2. the No-Fly list, 76 which lists persons prohibited from boarding aircraft; 
and 



"The Terrorist Screening Database is the central terrorist watchlist consolidated by the 
FBI's Terrorist Screening Center and used by multiple agencies to compile their specific 
watchlists and for screening. 

75 The other databases available to TSA are omitted because TSA considered them to be 
Sensitive Security Information. 

7C The No-Fly list is used to identify individuals who should be prevented from boarding an 
aircraft. The No-Fly and Selectee lists contain applicable records from the FBI's Terrorist 
Screening Center consolidated database of known or suspected terrorists. Pursuant to 
Homeland Security Presidential Directive 6, dated September 16, 2003, the Terrorist 
Screening Center — operational since December 2003 under the administration of the FBI — 
was established to develop and maintain the U.S. government's consolidated terrorist 
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3. the Terrorist Identity Datamark Environment terrorist list. 



TSA stated that the Transportation Security Operations Center checks 
passenger names submitted to it against these three databases if the 
passenger has been referred by a BDO to a LEO, but has not been arrested. 
Of the three databases that the Transportation Security Operations Center 
is to check in the case of a referral, passengers would have already been 
screened against two — the Selectee and No-Fly lists — in accordance with 
TSA passenger prescreening procedures when purchasing a ticket. The 
third database checked — the Terrorist Identity Datamark Environment — 
tracks terrorists but not persons wanted for other crimes. The FBI's NCIC 
information system would contain names of such persons, but is not 
among the three databases checked for nonarrest referrals. If the 
passenger has been arrested, the Transportation Security Operations 
Center will run the passenger's name against the additional law 
enforcement and intelligence databases available to it. 

In addition, TSA told us that the Operations Center does not have direct 
electronic access to the Terrorist Screening Database and must call the 
FBI's Terrorist Screening Center to provide it with a name to verify. TSA 
stated that this is done if a passenger's identity could not be verified using 
the Operations Center databases. In effect, if a passenger has been 
referred to a LEO, but not arrested, the Operations Center is to check the 
three databases shown above to verify the passenger's identity. If a 
passenger has been arrested, but the three databases do not list the 
person, the Center can check the additional databases available to it. If 
none of these databases can verify the person's identity, the Operations 
Center can contact the Terrorist Screening Center by telephone to request 
further screening. 



screening database (the watch list) and to provide for the use of watch-list records during 
security-related screening processes. See GAO-08-136T, Aviation Security: TSA Is 
Enhancing Its Oversight of Air Carrier Efforts to Screen Passengers against Terrorist 
Watch-List Records, but Expects Ultimate Solution to Be Implementation of Secure Flight 
(Washington, D.C.: Sept. 9, 2008). 

77 According to DHS, the Terrorist Identity Datamark Environment is the database 
maintained by the National Counterterrorism Center — the primary organization in the U.S. 
government for integrating and analyzing intelligence pertaining to terrorism and 
counterterrorism — to serve as a central repository for all information on known or 
suspected international terrorists with the exception of purely domestic terrorism 
information. See, DHS, Office of Inspector General, The DHS Process for Nominating 
Individuals to the Consolidated Terrorist Watchlist (Washington, D.C.: February 2008). 
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For passengers who have risen to the level of a LEO referral at an airport 
checkpoint, having the Transportation Security Operations Center 
consistently check their names against all the databases available to it 
could potentially help TSA identify threats to the aviation system and aid 
in "connecting the dots." TSA indicated that there are no obstacles to 
rapidly checking all databases rather than the three listed. We did not 
analyze the extent to which the law enforcement and intelligence 
databases available to TSA may contain overlapping information. 



TSA Lacks Program 
Effectiveness 
Measures for SPOT 
but Is Taking Steps to 
Improve Evaluation 
Capabilities 



TSA has established some performance measures by tracking SPOT 
referral and arrest data, but lacks the measures needed to evaluate the 
effectiveness of the SPOT program and, as a result, has not been able to 
fully assess SPOT'S contribution to improving aviation security. TSA 
emphasized the difficulty of developing performance measures for 
deterrence-based programs, but stated that it is developing additional 
measures to quantify the effectiveness of the program. The SPOT program 
uses teams to assess BDO proficiency, provide individual and team 
guidance, and address issues related to the interaction of BDOs with TSA 
checkpoint personnel. However, TSA does not systematically track the 
teams' recommendations or the frequency of the teams' airport visits. TSA 
states that it is working to address these issues and plans to do so by the 
end of fiscal year 2010. 



TSA Has Taken Action to 
Collect Data for Some 
Performance Measures, 
but Work Remains to 
Assess Progress Towards 
Achieving Strategic Goals 



TSA agreed that the SPOT program lacked sufficient performance 
measures in the past, but stated that it has some performance measures in 
place including tracking data on passengers referred for additional 
screening and the resolution of this screening, such as if prohibited items 
were found or if law enforcement arrested the passenger and the reason 
for the arrest. TSA is also working to improve its evaluation capabilities to 
better assess the effectiveness of the program. DHS's NIPP, internal 
controls standards, and our previous work on program assessment state 
that performance metrics and associated program evaluations are needed 
to determine if a program works and to identify adjustments that may 
improve its results. 78 Moreover, standard practices in program 
management for defining, designing, and executing programs include 



7S DHS, National Infrastructure Protection Plan: Partnering to Enhance Protection and 
Resiliency (Washington, D.C.: 2009); GAO/AIMD-00-2 1.3.1; and GAO, Performance 
Measurement and Evaluation: Definitions and Relationships, GAO-05-739SP 
(Washington, D.C.: May 2005). 
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developing a road map, or program plan, to establish an order for 
executing specific projects needed to obtain defined programmatic results 
within a specified time frame. 79 Congress also needs information on 
whether and in what respects a program is working well or poorly to 
support its oversight of agencies and their budgets; and agencies' 
stakeholders need performance information to accurately judge program 
effectiveness. 80 For example, in the Senate Appropriations Committee 
report accompanying the fiscal year 2010 DHS appropriations bill, 81 the 
committee noted that while TSA has dramatically increased the size and 
scope of SPOT, resources were not tied to specific program goals and 
objectives. In addition, the conference report accompanying the fiscal 
year 2010 DHS appropriations act requires TSA to report to Congress, 
within 60 days of enactment, on the effectiveness of the program in 
meeting its goals and objectives, among other things. 82 This report was 
completed on March 15, 2010. 

Although TSA tracks data related to SPOT activities including prohibited 
items, law enforcement arrests related to SPOT referrals, and reasons for 
the arrests (output measures), it has not yet developed measures to gauge 
SPOT's effectiveness in meeting TSA strategic goals (outcome measures), 
such as identifying individuals who may pose a threat to the transportation 
system. 83 OMB encourages the use of outcome measures because they are 
more meaningful than output measures, which tend to be more process- 
oriented or means to an end. 84 For example, TSA's Assistant General 



73 The Project Management Institute, The Standard for Program Management© (2006). 

s °GAO, Executive Guide: Effectively Implementing the Government Performance and 
Results Act, GAO/GGD-96-118 (Washington, D.C.: June 1996). 

81 See S. Rep. No. 111-31, at 56-57 (2009); see also S. Rep. No. 110-396, at 59 (2008). 

82 See H.R. Rep. No. 111-298, at 77 (2009) (Conf. Rep.). The report further directs that GAO 
review the report submitted by TSA and provide its findings to the committees no later 
than 120 days after the SPOT report is submitted to the committees. 

83 Output measures help determine the extent to which an activity was performed as 
planned. Outcome-related measures are more robust measures because they provide a 
more comprehensive assessment of the success of the agency's efforts, as stated in DHS's 
2009 NIPP. 

S4 OMB and the Council for Excellence in Government, Performance Measurement 
Challenges and Strategies (Washington, D.C.: June 18, 2003). 
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Manager for the Office of Operation Process and Performance Metrics 
told us that SPOT staffing levels are currently used as one performance 
metric. The official said that since the SPOT is an added layer of security, 
additional SPOT staffing would add to security effectiveness. While 
staffing levels may help gauge how fast the program is growing, they do 
not measure the effectiveness in meeting strategic goals. 

Similarly, TSA also cited the number of prohibited items discovered by 
BDOs in SPOT metrics reports as a measure of program success. 86 
However, TSA told us that possession of a prohibited item is often an 
oversight and not an intentional act; moreover, other checkpoint screening 
layers are intended to find such items, such as the TSOs and the property 
screening equipment. 87 TSA also cited measures of BDO job performance 
as some of the existing measures of program effectiveness, but noted that 
these are "pass/fail" assessments of individual BDOs, rather than overall 
program measures. 

TSA notes that one purpose of the SPOT program is to deter terrorists, but 
that proving that it has succeeded at deterring terrorists is difficult 
because the lack of data has presented challenges for the SPOT program 
office when developing performance measures. We agree that developing 
performance measures, especially outcome measures, for programs with a 
deterrent focus is difficult. Nevertheless, such measures are an important 
tool to communicate what a program has accomplished and provide 
information for budget decisions. TSA uses proxy measures — indirect 
measures or indicators that approximate or represent the direct 
measure — to address deterrence, other security goals, or a combination of 
both. For example, TSA tracks the number of prohibited items found and 
individuals arrested as a result of SPOT referrals. According to OMB, 
proxy measures are to be correlated to an improved security outcome, and 
the program should be able to demonstrate — such as through the use of 



85 The Office's primary work involves metrics infrastructure; it assists TSA programs, if 
requested, in developing applications to track quantitative measures, such as surrendered 
items. It also tracks data for its Management Objectives Report related to three areas: 
employees, security effectiveness, and efficiency. 

8f The types of prohibited items found have included knives, guns, gun ammunition, certain 
chemicals, strike-anywhere matches, and certain liquids/gels/aerosols; other illegal items 
discovered include narcotics and fraudulent identity documents. 

"'According to TSA, TSOs focus on detecting high-risk threats which have the ability to 
cause catastrophic damage to an airplane in flight (e.g., explosives). 
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modeling — how the proxies tie to the eventual outcome. In using a 
variety of proxy measures, failure in any one of the identified measures 
could provide an indication on the overall risk to security. However, 
developing a plan that includes objectives, milestones, and time frames to 
develop outcome-based performance measures could better position TSA 
in assessing the effectiveness of the SPOT program. 

With regard to more readily quantifiable output performance measures, 
such as the number of referrals by BDOs, or the ratio of arrests to 
referrals, TSA was limited in its ability to analyze the data related to these 
measures. The SPOT database includes information on all passengers 
referred by BDOs for additional SPOT screening including the behaviors of 
the passengers that led to the additional screening, as well as the 
resolution of the screening process (e.g., no further action taken, law 
enforcement notification, law enforcement investigation, arrested, and 
reason for arrest). However, TSA reported that any analysis of the data 
had to be done manually. 89 

In March 2010, TSA migrated the SPOT referral data to its Performance 
Management Information System, allowing for more statistical and other 
analyses. According to TSA, migrating the SPOT referral database will 
enhance the SPOT program's analytic capabilities. For example, TSA 
stated that it would be able to conduct trend analyses, better segregate 
data, and create specific reports for certain data. This includes better 
tracking of performance data at specific airports, analyzing by categories 
of airports (threat or geographic location), and tracking the performance 
data of individual BDOs, such as number of referrals, number of arrests, 
arrest to referral ratios, and other analyses. However, since these changes 
to the database were not complete at the time of our audit, we could not 
assess whether the problems we identified with the database had been 
corrected. 



88 OMB and the Council for Excellence in Government, Performance Measurement 
Challenges and Strategies (Washington, D.C.: June 18, 2003). 

89 We also found that the SPOT referral database had a number of weaknesses. TSA 
designated our discussion of these weakness as sensitive security information. 
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Over 4 Years, SPOT Resulted in 
About 1,100 Arrests Out of 
Almost 14,000 Referrals to Law 
Enforcement 



The SPOT referral database records the total number of SPOT referrals 
since May 29, 2004, how many were resolved, how many passengers BDOs 
referred to LEOs, the recorded reasons for the referral, and how many 
referrals led to arrests, among other things. As shown in figure 4, we 
analyzed the SPOT referral data for the period May 29, 2004, to August 31, 
2008. 



Figure 4: Passenger Boardings at SPOT Airports, May 29, 2004, through August 31 , 
2008 
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Source: G AO analysis of TSA and Bureau of Transportation Statistics data. 

Note: Figure 4 is not drawn to scale. 
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Figure 4 shows that approximately 2 billion passengers boarded aircraft at 
SPOT airports from May 29, 2004, through August 31, 2008. 90 Of these, 
151,943 (less than l/100 th of 1 percent) were sent to SPOT referral 
screening, and of these, 14,104 (9.3 percent) were then referred to LEOs. 
These LEO referrals resulted in 1,083 arrests, or 7.6 percent of those 
referred, and less than 1 percent of all SPOT referrals (0.7 percent of 
151,943). 



We also analyzed the reasons for arrests resulting from SPOT referrals, for 
the May 29, 2004, through August 31, 2008, period. Table 2 shows, in 
descending order, the reasons for the arrests. 



Table 2: Reasons for Arrests from SPOT Referrals, May 29, 2004 through August 
31,2008 


Reasons for arrest 


Number 


Percentage 


Illegal alien 


427 


39 


Outstanding warrants 


209 


19 


Possession of fraudulent documents 


166 


15 


Other 


128 


12 


Possession of suspected drugs 


125 


12 


No reason given 


16 


1 


Undeclared currency 


8 


1 


Suspect documents 


4 





Total 


1,083 


99 a 



Source: TSA, SPOT referral database from period of May 29, 2004, through August 31 , 2008. 

"Total does not add to 100 percent due to rounding. 



While SPOT personnel did not determine a specific reason for arrest for 
128 cases categorized as "other" or 16 other cases categorized as "no 
reason given," our analysis of the SPOT database found that a specific 
reason for arrest could have been determined for these cases by using the 
LEO resolution notes included in the database. For example, we identified 
43 additional arrests related to fraudulent documents, illegal aliens, and 
suspect documents, among others. The remaining 101 arrests originally 
characterized as "other" or "no reason given" included arrests for reasons 



9C Our estimate of the total number of passengers who went through checkpoints is based 
on Bureau of Transportation Statistics data that we obtained for the airports at which 
SPOT was deployed during this period. Some figures were rounded. 
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such as intoxication, unruly behavior, theft, domestic violence, and 
possession of prohibited items. Many of the arrests resulting from BDO 
referrals would typically fall under the jurisdiction of various local, state, 
and federal agencies and are not directly related to threats to aviation 
security. For example, the 427 individuals arrested as illegal aliens, and 
the 166 arrested for possession of fraudulent documents, are subject to the 
enforcement responsibilities shared by U.S. Immigration and Customs 
Enforcement (ICE) and CBP. Although outstanding warrants and the 
possession of fraudulent or suspect documents could be associated with a 
terrorist threat, TSA officials did not identify any direct links to terrorism 
or any threat to the aviation system in any of these cases. 

According to TSA, anecdotal examples of BDO actions at airports show 
the value added by SPOT to securing the aviation system. However, 
because the SPOT program has not been scientifically validated, it cannot 
be determined if the anecdotal results cited by TSA were better than if 
passengers had been pulled aside at random, rather than as a consequence 
of being identified for further screening by BDOs. Some of the incidents 
cited by TSA include the following. 

• A BDO referred two passengers who were traveling together to referral 
screening due to suspicious behavior. During secondary screening, one 
passenger presented fraudulent travel documents. The other could not 
produce any documentation of his citizenship and it was determined he 
was in the United States illegally. ICE responded and interviewed both 
passengers. ICE stated one passenger was also in possession of 
$10,000 dollars which alarmed positive for narcotics when swept by a 
K-9 team. ICE arrested one passenger on a federal charge of 
possession of fraudulent identification documents and entry without 
inspection. ICE stated charges are still pending for the possession of 
$10,000. The second passenger was charged with a federal charge of 
entry without inspection. 

• A BDO referred a passenger to referral screening for exhibiting 
suspicious behavior. Port Authority of Portland (Oregon) Police 
responded and interviewed the passenger who did not give a statement. 
LEOs conducted an NCIC check which revealed that there was an 
outstanding warrant for the failure to appear for a theft charge. LEOs 
arrested the passenger on a state charge for an outstanding warrant for 
the failure to appear for theft. 

• A BDO referred a passenger for referral screening due to suspicious 
behavior. During the referral, the passenger admitted that he was 
unlawfully present in the United States. The Orlando (Florida) Police 
Department and CBP responded and interviewed the passenger who 
stated he had $100,000 in his checked baggage, which was confirmed 
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by CBP. The passenger was arrested on a federal charge of illegal 
entry. 

Because these are anecdotal examples, they cannot be used to reliably 
generalize about the SPOT program's overall effectiveness or success rate. 
Our analysis of the SPOT referral database found that the referral data do 
not indicate if any of the passengers sent to referral screening, or those 
arrested by LEOs after being referred to them, intended to harm the 
aircraft, its passengers, or other components of the aviation system. 
Additionally, SPOT officials told us that it is not known if the SPOT 
program has ever resulted in the arrest of anyone who is a terrorist, or 
who was planning to engage in terrorist-related activity. 



Reviewing Airport Video 
Recordings of Individuals Later 
Arrested or Who Pleaded Guilty 
for Engaging in Terrorism- 
Related Activities Could Help 
TSA Better Identify Terrorist- 
Related Behaviors 



Studying airport video recordings of the behaviors exhibited by persons 
waiting in line and moving through airport checkpoints and who were later 
charged with or pleaded guilty to terrorism-related offenses could provide 
insights about behaviors that may be common among terrorists or could 
demonstrate that terrorists do not generally display any identifying 
behaviors. TSA officials agreed that examining video recordings of 
individuals who were later charged with or pleaded guilty to terrorism- 
related offenses, as they used the aviation system to travel to overseas 
locations allegedly to receive terrorist training or to execute attacks, may 
help inform the SPOT program's identification of behavioral indicators. In 
addition, such images could help determine if BDOs are looking for the 
right behaviors or seeing the behaviors they have been trained to observe. 

Using CBP and Department of Justice information, we examined the travel 
of key individuals allegedly involved in six terrorist plots that have been 
uncovered by law enforcement agencies. 91 We determined that at least 16 
of the individuals allegedly involved in these plots moved through 8 
different airports where the SPOT program had been implemented. 92 Six 



91 The analysis included only flights leaving the United States. Department of Justice data 
show that more than 400 individuals have been convicted in the United States for 
terrorism-related offenses since September 11, 2001. We did not examine the travel 
itineraries of all these individuals. 

92 The events included the Mumbai, India attack of 2008; a plot to attack the Quantico, 
Virginia, Marine base in 2008; an effort by five Americans to receive training and fight in 
Pakistan in December 2009; a plot to attack infrastructure in New York City in 2009; an 
effort to provide men and support for terrorists in Somalia in 2008; and an attack on a U.S. 
base in Afghanistan by an American who received training in Pakistan. We were unable to 
confirm whether BDOs were stationed at the checkpoints used by these individuals at the 
time they traveled. 
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of the 8 airports were among the 10 highest risk airports, as rated by TSA 
in its Current Airport Threat Assessment. In total, these individuals moved 
through SPOT airports on at least 23 different occasions. For example, 
according to Department of Justice documents, in December 2007 an 
individual who later pleaded guilty to providing material support to Somali 
terrorists boarded a plane at the Minneapolis-Saint Paul International 
Airport en route to Somalia to join terrorists there and engage in jihad. 
Similarly, in August 2008 an individual who later pleaded guilty to 
providing material support to Al-Qaeda boarded a plane at Newark Liberty 
International Airport en route to Pakistan to receive terrorist training to 
support his efforts to attack the New York subway system. 

Our survey of Federal Security Directors at 161 SPOT airports indicated 
most checkpoints at SPOT airports have surveillance cameras installed. 
As we previously reported, best practices for project management call for 
conducting feasibility studies to assess issues related to technical and 
economic feasibility, among other things. 93 In addition, Standards for 
Internal Control state that effectively using available resources is one 
element of functioning internal controls. 94 TSA may be able to utilize the 
installed video infrastructure at the nation's airports to study the behavior 
of persons who were later charged with or pleaded guilty to terrorism- 
related offenses, and determine whether BDOs saw the behaviors. The 
Director of Special Operations in TSA's Office of Inspection told us that 
video recordings could be used as a teaching tool to show the BDOs which 
behaviors or activities they did or did not observe. In addition, TSA 
indicated that although the airports may have cameras at the security 
screening checkpoints, the cameras are not owned by TSA, and in many 
cases, they are not accessible to TSA. However, TSA officials lack 
information on the scope of these potential limitations because prior to 
our work TSA did not have information on the number of checkpoints 
equipped with video surveillance. We obtained this information as part of 
our survey of Federal Security Directors at SPOT airports. While TSA 
officials noted several possible limitations of the use of the existing video 
surveillance equipment, these images provide TSA a means of acquiring 
information about terrorist behaviors in the checkpoint environment that 



93 See, GAO, Supply Chain Security: Feasibility and Cost-Benefit Analysis Would Assist 
DHS and Congress in Assessing and Implementing the Requirement to Scan 100 Percent 
ofU.S.-Bound Containers, GAO-10-12 (Washington, D.C.: Oct. 30, 2009). The Project 
Management Institute, A Guide to the Project Management Body of Knowledge. 

M GAO/AIMD-00-2 1.3.1. 
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is not available elsewhere. If current research determines that the SPOT 
program has a scientifically validated basis for using behavior detection 
for counterterrorism purposes in the airport environment, then conducting 
a study to determine the feasibility of using images captured by video 
cameras could better position TSA in identifying behaviors to observe. 



TSA sends standardization teams to SPOT airports on a periodic basis to 
conduct activities related to quality control. Teams observe SPOT 
operations at an airport for several days, working side by side with the 
BDOs, on multiple shifts, observing their performance, offering guidance, 
and providing training when required. According to TSA, the purpose of a 
standardization team visit is to provide operational support to the BDOs, 
which includes additional training, mentoring, and guidance to help 
maintain a successful SPOT program. 

The standardization teams are comprised of at least two G-Band, or 
Expert 95 BDOs who have received an additional week of training on SPOT 
behaviors and mentoring skills. SPOT officials stated that the SPOT 
program uses its standardization teams to assess overall BDO proficiency 
by observing BDOs, reviewing SPOT score sheet data, and other relevant 
data. Standardization teams may also provide a Behavior Observation and 
Analysis review class to refresh BDOs if the team determines that such a 
class is needed. The SPOT program director also said that the 
standardization teams aim to monitor the airport's compliance with the 
SPOT program's Standard Operating Procedures. As part of this 
mentoring approach, the standardization teams provide individual and 
team guidance to the BDOs, offer assistance in program management, and 
cover issues related to the interaction of BDOs with other TSA checkpoint 
personnel. 

TSA reported to us that it does not systematically track the 
standardization teams' recommendations or the frequency of the teams' 
airport visits. Standards for Internal Control state that programs should 
have controls in place to assess the quality of performance over time and 
ensure that the findings of audits and other reviews are promptly resolved. 
Managers are to (1) promptly evaluate findings from audits and other 
reviews, including those showing deficiencies and recommendations 



95 G-Band, or Expert BDOs, have advanced to a lead role, are able to provide technical 
expertise on the SPOT program, and are one band away from a supervisory role. 



Standardization Teams 
Assess BDO Proficiency in 
SPOT Activities and 
Provide Guidance and 
Mentoring to BDOs 
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reported by auditors and others who evaluate agencies' operations; (2) 
determine proper actions in response to findings and recommendations 
from audits and reviews; and (3) complete, within established time frames, 
all actions that correct or otherwise resolve the matters brought to 
management's attention. 96 Although the standardization teams may 
provide an airport Federal Security Director with recommendations on 
how to improve SPOT operations, the SPOT program director stated that 
Federal Security Directors are not required to document whether they 
have implemented the team recommendations. TSA officials told us that 
standardization teams can follow up on recommendations made during 
previous visits. However, TSA did not track whether corrective actions 
were implemented or the frequency of the team's airport visits to ensure 
the implementation of the airport's SPOT program. TSA officials stated 
that they are currently examining ways to compile data to address this 
issue, and expect to have a system in place in fiscal year 2010. 



TSA Developed and 
Deployed SPOT 
Training but Further 
Action Could 
Enhance Its 
Effectiveness 



Although TSA has taken steps to incorporate all four elements of an 
effective training program by planning, designing, implementing, and 
evaluating training for BDOs, further action could help enhance the 
training's effectiveness. TSA initially consulted outside experts for help in 
the training's development, which began as a half-day course and has 
grown to include classroom, on-the-job, and advanced training. TSA also 
has efforts underway to improve its training program, such as the 
deployment of SPOT recurrent training. However, TSA evaluations of 
SPOT program instructors found mixed quality among them, from 2006 
onwards. Additionally, TSA has ongoing plans to evaluate the SPOT 
training for effectiveness, but has not yet developed time frames and 
milestones for completing the evaluation. 



TSA Has Taken Actions to 
Develop and Deploy SPOT 
Training 

TSA's SPOT Training Evolved m 2003, TSA officials at Boston Logan International airport developed the 
Over Time initial half-day training course for SPOT based on an existing course 

developed for the Massachusetts State Police. Their goal was to take the 



'GAO/AIMD-00-21.3.1. 
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behavior detection program designed for law enforcement and apply it to 
screeners at airport checkpoints. According to TSA officials at Boston 
Logan, after they recognized that the lecture-style course they originally 
designed was not effective, they tasked an instructional system designer 
from TSA's Workplace Performance and Training (the former name of 
TSA's Operational and Technical Training Division) 97 and an industrial 
psychologist from the Office of Human Capital to redesign and expand the 
course, which was piloted in 2005. The 2007 SPOT strategic plan included 
training objectives for the SPOT program as follows: 

• reviewing existing behavior observation training providers, 

• establishing and prioritizing multimodal training and assistance efforts 
based on threat assessments and critical infrastructure, 

• establishing a Center of Excellence for Behavior Detection Program 
training that would continually enhance the quantity and quality of 
training to selected candidates, and 

• developing a recurrent training program designed to refresh and hone 
skills needed for an effective Behavior Detection Program. 

Since that time, the SPOT program implemented, or is in the process of 
implementing, some of these objectives. For example, in 2008, as part of 
its effort towards establishing a center for excellence in behavior 
detection training (third objective), the SPOT program participated in a 
meeting with behavior detection training officials from various DHS 
components facilitated by DHS's Screening Coordination Office to 
promote the sharing of information about behavior detection training and 
foster future collaboration. Additionally, the SPOT program worked with 
TSA's Operational and Technical Training Division to create a recurrent 
training component for BDOs (fourth objective). For example, in 2008, the 
SPOT program office added a course on detecting microfacial expressions 
called Additional Behavior Detection Techniques. 98 This 3-day course 
builds on the behavior detection skills taught in basic training, by teaching 



9 TSA's Operational and Technical Training Division, within the Office of Security 
Operations, provides assistance with development and implementation of technical 
training for screening, Behavior Detection Officers, Bomb Appraisal Officers, the Aviation 
Direct Access Screening Program, and technical management training. 

98 In May 2009, the title of the course was changed to "Additional Behavior Detection 
Techniques" because ABDT is actually a supplemental tool for BDOs to use during the 
Casual Conversation phase of SPOT Referral Screening. The course was formerly titled 
"Advanced Behavior Detection Techniques." Microfacial expressions are very brief facial 
expressions that can last as little as 1/25 of a second. 
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BDOs how to detect microfacial expressions. After pilot testing, the 
course began implementation nationwide in January 2009. 

TSA Consulted with Some m developing an effective training program, we previously reported that 

Experts on Developing SPOT consultation with subject matter experts and expert entities is a core 
Training characteristic of the strategic training and development process." TSA 

SPOT program staff told us that they consulted with experts on behavior 
detection and observed existing behavior detection courses before 
deploying the SPOT training program. According to SPOT program 
officials, a TSA staff member from Boston Logan International Airport 
attended other training programs offered by other federal agencies and 
private training organizations to inform the design of SPOT training. 100 
TSA officials told us that information from the training courses was used 
to help develop the list of behaviors or "stress elevators" for the program, 
and that the point system used to identify passengers for referral screening 
was based in part on consultations with several subject-matter experts. 

TSA documentation also notes that a SPOT working group created in 
February 2004 consulted with the FBI's Behavioral Science Unit. 101 The 
Behavioral Science Unit specializes in developing and facilitating training, 
research, and consultation in the behavioral sciences for the FBI, law 
enforcement, intelligence, and military communities. While TSA officials 
from Boston Logan told us that the FBI was included in this initial SPOT 
working group, these officials agree that coordination with the FBI lapsed 
until June 2009 when the SPOT Program Office reengaged with the 
Behavioral Science Unit, and held a meeting with the unit at the FBI 
Academy in Quantico, Virginia. Since that meeting, a subject matter 
expert from the SPOT Program Office has been invited to be a member of 



GAO, Human Capital: A Guide for Assessing Strategic Training and Development 
Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: Mar. 1, 2004). 

'""The TSA staff member attended the following external training courses: John Reid and 
Associates' Reid Techniques of Interrogation and Advanced Reid Techniques of 
Interrogation; Massachusetts State Police Academy's Basic Investigations and Professional 
Development Program Interview Techniques; International Security Defense Systems' 
Verification Agent for Virgin Atlantic Security Systems; New Mexico Technology, Materials 
and Research Center's Prevention and Response to Suicide Bomber Indicators; Abraxis 
Corporation's Detecting Deception and Eliciting Response; Langevin Learning Services' 
Instructional Techniques for New Instructors; Ekman Group's Understanding Emotions 
and Detecting Truth; Chameleon Associates' Suspicious Behavior Detection; and Federal 
Transit Administration's Terrorist Awareness, Recognition, and Response. 

101 The purpose of the SPOT working group was to help refine the list of SPOT behaviors and 
to develop standard operating procedures and a concept of operations for the program. 
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the Terrorism Research and Analysis Project, which is an ongoing working 
group sponsored by the unit. 

In July 2008, DHS's Screening Coordination Office facilitated a 
collaborative discussion on behavior detection that included TSA, CBP, 
and Secret Service officials to better ensure that components within DHS 
share information regarding their efforts in behavior detection and provide 
a forum for components to have an informed and collaborative discussion 
on current capabilities, best practices, and lessons learned. According to 
TSA, no further contact has occurred between the DHS Behavior 
Detection Working Group and the SPOT program. Thus, the extent to 
which the working group's expertise will be used to refine or augment 
SPOT training in the future is not yet clear. 



SPOT Program Office 
Recently Deployed 
Recurrent Training 



Along with basic and remedial training required by the Aviation and 
Transportation Security Act, TSA policy requires its screening force to 
regularly complete recurrent (refresher) training. TSA recognized that 
ongoing training of screeners on a frequent basis and effective supervisory 
training are critical to maintaining and enhancing skills learned during 
basic training. According to agency officials, TSA is currently working 
with DHS S&T to determine the necessary frequency for refresher training 
for each training course within the SPOT program. Furthermore, TSA 
plans to place BDOs under TSA's Performance and Accountability 
Standards System (PASS) beginning in fiscal year 2010. This will include a 
recertification module. 



In 2008, the SPOT program office began the process for developing 
recurrent SPOT training. Our internal control standards and training 
assessment guidance suggest that such refresher training should be 
considered integral to an effective training program from the start because 
work conditions and environments can be expected to change over time, 
and additional or updated training is essential to ensuring that the program 
mission continues to be accomplished. 102 According to the SPOT program 
office, the recently deployed recurrent training will be semiannual. TSA's 
Operational and Technical Training Division initially planned to pilot test 
recurrent training in April 2009 followed by full implementation of the 
course in approximately May 2009. Because the Operational and 
Technical Training Division focus was shifted to completing the revisions 



102 GAO/AIMD-00-21.3.1 and GAO-04-546G. 
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for the SPOT basic certification course, recurrent training was delayed 
until September 2009 when they released the training on TSA's Online 
Learning Center. 



Instructor Evaluations 
Found Mixed Quality; 
Issues with Program 
Management Led to 
Instructor Retraining 



Our previous work on elements of effective training states that instructors 
must be both knowledgeable about the subject matter and issues involved, 
as well as able to effectively transfer these skills and knowledge to 
others. 103 Moreover, internal control standards state that all personnel 
need to possess and maintain a level of competence that allows them to 
accomplish their assigned duties. 104 Management needs to identify 
appropriate knowledge and skills needed for various jobs and provide 
needed training, as well as to ensure that those teaching the skills are 
themselves competent. 

TSA conducted internal assessments of SPOT instructors episodically 
from 2006 through March 2008. These assessments involved a few 
instructors being rated at a time, and found a wide range of competency 
among the instructors. In January 2009, TSA's Office of Inspections and 
Investigations began an investigation of the SPOT training manager, who 
resigned shortly thereafter. TSA investigators determined that the training 
manager and other trainers had created a hostile training environment that 
intimidated some trainees. To address this problem, TSA stated that the 
program office reexamined the SPOT training program nationally. This 
included recertifying 47 of 54 SPOT instructors in March 2009, which 
included evaluation by TSA's Office of Human Capital, Quality Assurance 
assessors. Additionally, in July 2009, TSA centralized SPOT training at five 
permanent, regional training facilities in Orlando, Florida; Houston, Texas; 
Phoenix, Arizona; Denver, Colorado; and Philadelphia, Pennsylvania. 105 
According to the SPOT program director, this will allow the SPOT 
program office more oversight over training. Previously, training was 
provided at individual airports. 



103 GAO-04-546G. 
104 GAO/AIMD-00-21.3.1. 

105 The SPOT program retains the discretion to train BDOs at a site other than one of the five 
training facilities if it is more fiscally responsible to do so. For example, if there are 15 
BDO candidates at a single airport, the SPOT program will train them at that airport rather 
than sending them to a training facility. 
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After the March 2009 recertification training, ratings scores of SPOT 
instructors showed less variation than did previous ratings. We reviewed 
the quality assurance instructor evaluations of two SPOT instructors 
conducted by TSA's Office of Human Capital, Training Standards and 
Evaluation Branch, and the 167 SPOT program instructor evaluations of 54 
SPOT instructors conducted by the SPOT program office and TSA's 
Operational and Technical Training Division since the program started in 
October 2006. 106 After the recertification training, 93 percent of instructors 
were rated as exceeding expectations, compared to 30 percent in the 2006 
to September 2008 ratings. Table 3 shows the ratings of instructors for 
March 2009 compared to the period of 2006 to September 2008. 107 



Table 3: SPOT Instructor Evaluation Ratings, 2006 to September 2008, and March 2009 


Number of 
instructor 


Unsatisfactory 
(0-74%) 


Needs 
improvement 
(75-84%) 


Meets 
expectations 
(85-94%) 


Exceeds 
expectations 
(95-100%) 


No numeric 
score given 


evaluations 


Number Percent 


Number Percent 


Number Percent 


Number Percent 


Number Percent 


2006- 
Sept 

2008 73 


3 4% 


5 7% 


36 49% 


22 30% 


7 10% 


March 

2009 94 


0% 


1 1% 


6 6% 


87 93% 


0% 



Source: GAO analysis of TSA Quality Assurance Instructor Evaluations for SPOT. 



In addition to the variation in numeric scores and rating levels for the 2006 
to September 2008 period, as shown in table 3, we found substantial 
variation in the comments about instructor competency for the same 



Some SPOT instructors have been evaluated multiple times. While the SPOT program 
office provided us with print or electronic copies of all SPOT instructor evaluations, some 
forms contained only numeric ratings and no written comments; others had no numeric 
scores. Because instructor names were redacted from the evaluations, the numbers may 
include duplicates. Additionally, the evaluations containing written comments were not 
always filled out using complete sentences, making it difficult to ascertain the rater's 
assessment of the instructor. 

'"'SPOT Instructors are evaluated using a Quality Assurance Instructor Evaluation, TSA 
Form 1909. Using this form, the evaluator assigns either (zero) points, 0.5 points, or 1 
point for each of 57 ratable items depending on whether the instructor meets the standard 
as written, needs improvement to meet the standard, or does not meet the standard. The 
total points are then entered into a formula that generates a percentage. This percentage is 
used to determine the overall rating. Instructors receiving a score of 95 percent to 100 
percent are rated as exceeds expectations; 85 percent to 94 percent are rated as meets 
expectations; 75 percent to 84 percent are rated as needs improvement; and percent to 74 
percent are rated as unsatisfactory. 
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period. For example, in 32 out of 74 instructor evaluation forms that we 
reviewed where comments were made about the instructor prior to 2009, 
the comments ranged from superb to needs more experience as an 
instructor, as well as needs more time performing the job as a BDO to be 
able to teach others. In the comments on an instructor who was rated as 
"meets expectations," the instructor was described as having "limited 
experience within the SPOT program," that this was "a major concern," 
and it was recommended that the instructor spend as much time as 
possible functioning as a BDO. In other cases, however, SPOT instructors 
were described as competent, solid, and outstanding. For example, one 
instructor who received a rating of "exceeds expectations" was described 
as a superb instructor who "is a valued member of the National Training 
Team." As noted above, following the March 2009 recertification training, 
93 percent of the instructors received a rating of "exceeds expectations" 
with only 1 percent "needing improvement." Of the 94 instructor 
evaluations completed in March 2009, 82 contained written comments. Of 
these, multiple SPOT instructors were described as excellent, 
knowledgeable, and effective. For example, an instructor who received a 
rating of "exceeds expectations" was noted as demonstrating a high degree 
of material knowledge and great presentation skills. TSA attributed the 
increase in instructor ratings to two factors. The first is low turnover 
among SPOT instructors, which allows instructors to hone both their 
technical and instructor skills. The second factor cited by TSA is that TSA 
conducted a 2-day instructor refresher training immediately prior to the 
evaluations in March 2009. To ensure all instructors were reevaluated 
within a specific time frame, evaluations were scheduled and conducted in 
a controlled environment. Instructors knew in advance they were going to 
be evaluated and delivered modules of the BDO certification course to 
other BDO instructors. 



TSA Has Taken Some 
Action, but Has Not 
Evaluated the SPOT 
Training Program for 
Effectiveness 



We previously reported that evaluation is an integral part of training and 
development efforts, and that agencies need to systematically plan for and 
evaluate the effectiveness of training and development. 108 Employing 
systematic monitoring and feedback processes can help by catching 
potential problems at an early stage, thereby saving valuable time and 
resources that a major redesign of training would likely entail. Similarly, 
in 2006, TSA's Operational and Technical Training Division issued general 
evaluation standards for training programs, stating that training programs 



10S GAO-04-546G. 
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should be comprehensively evaluated on a periodic basis to identify 
program strengths and weaknesses. 109 Moreover, standard practices in 
program management for defining, designing, and executing programs 
include developing a road map, or program plan, to establish an order for 
executing specific projects needed to obtain defined programmatic results 
within a specified time frame. 110 

The former SPOT training manager told us that the SPOT program 
internally evaluates the effectiveness of SPOT training through the job 
knowledge tests that BDO candidates must pass following the classroom 
portion of the training and the SPOT Proficiency/On-the-Job Training 
Checklist following the on-the-job portion of the training. Furthermore, 
the former training manager told us that TSA knows that the SPOT training 
is effective because BDOs are able to recognize behaviors at the 
checkpoint, and because of BDOs' demonstrated ability to identify 
criminals — such as drug couriers or people with outstanding arrest 
warrants — through the screening process. 

Although TSA has not conducted a comprehensive analysis of the 
effectiveness of the SPOT training program, TSA's Office of Human 
Capital, Training Standards and Evaluation Branch conducted training 
evaluations to assess how students use what they were taught in the SPOT 
basic training course. Specifically, from July through September 2008, the 
Training Standards and Evaluation Branch conducted evaluations at 5 of 
the 161 airports where the SPOT program is currently operating. Based on 
BDO feedback at the 5 airports, the Training Standards and Evaluation 
Branch's final report contained a series of recommendations for improving 
the SPOT training program. These recommendations and TSA's actions to 
address them are summarized in table 4. 



'TSA, Operational and Technical Training Division, Training Standards (Sept. 28, 2006). 
The Project Management Institute, The Standard for Program Management© (2006). 
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Table 4: TSA Training Standards and Evaluation Branch Recommendations for 
Improving SPOT Training and TSA Actions on the Recommendations 



Training Standards and Evaluation 

Branch recommendations TSA action on recommendations 

Ensure training instructors adhere to a set TSA sent 47 TSA Approved Instructors for 
of professional guidelines. the SPOT program to recertification training 

in March 2009. 

Add local policies and procedure as an No action. 3 
addendum to the (SPOT) Training. 

Include more role-playing and scenarios in TSA added more role-playing scenarios to 
the classroom training so all trainees can their basic SPOT training, 
practice casual conversation skills. 

Develop recurrent training that can be TSA developed and deployed recurrent 
placed on the TSA Online Learning Center, training on the TSA Online Learning Center 

in September 2009. 

Develop templates for writing reports. TSA added an Incident Report Writing 

course to the TSA Online Learning Center. 
Additionally, TSA has developed templates 
for Incident Reports and After Action 
Reports. TSA has also developed Online 
Learning Center training for completing 
SPOT Referral Reports. 

Provide more real world videos. TSA revised the SPOT training videos in 

late 2008. 

Provide recurrent training of behaviors The video scenarios for recurrent training 
through online videos. will be available in the second quarter of 

fiscal year 2010. 

Add parts of the Bomb Appraisal Officer No action. 3 
task into the training. 

Provide recurrent training outside of TSA No action. 3 
(more Immigration and Customs 
Enforcement, DEA, and CBP training). 

Have BDOs spend more time with an On- No action. 3 
the-Job-Training mentor. 

Validate the training for course content and In 2009, in coordination with DHS S&T, 
On-the-Job-Training. TSA began the scientific analysis of the 

BDO position to empirically derive and 
validate the knowledge, skills, and 
attributes that it requires. The analysis is 
projected to be completed in fiscal year 
2010. 

Clarify SPOT'S "Walk-the-Line" policy and TSA issued revised SPOT Standard 
communicate it to all BDO personnel. Operating Procedures to all BDOs in 

January 2009. 



Source: TSA, Training Standards and Evaluation Branch, Office of Human Capital, Memorandum For Operational and Technical 
Training, and Behavior Detection and Document Validation Branch, Office of Security Operations on Training Transfer (L3) of SPOT 
Training, October 30, 2008. 
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"According to TSA, the SPOT program office will determine if the recommended action is appropriate 
after the BDO job task analysis and training task analysis are completed. 

Additionally, in conjunction with S&T, TSA conducted a training 
effectiveness evaluation on the Additional Behavior Detection Techniques 
course, which showed a statistically significant increase in knowledge and 
skills following completion of the course. 

S&T is currently conducting a BDO job task analysis, which may be used 
to evaluate and update the SPOT training curriculum. Following the 
completion of the job task analysis — anticipated in mid-May 2010 — TSA's 
Operational and Technical Training Division intends to conduct an in- 
depth training gap analysis, 111 which will take approximately 2 months to 
complete. Following completion of the training gap analysis, the agency 
will develop project plans, including milestones for future development 
efforts, to address any training concerns. However, to date, the agency 
does not have an evaluation plan including time frames and milestones for 
completion. According to the Operational and Technical Training 
Division, TSA will conduct periodic evaluations as the BDO position 
evolves. By conducting a comprehensive evaluation of the effectiveness of 
its training program, TSA will be in a better position to determine if BDOs 
are being taught the knowledge and skills they need to perform their job. 
Furthermore, by developing milestones and time frames for conducting 
such evaluations systematically, as well as on a periodic basis, TSA could 
help ensure that the SPOT training program is evaluated in accordance 
with its directives to help ensure that the program continues to provide 
BDOs with the necessary tools required to carry out their responsibilities. 



TSA developed the SPOT program in the wake of September 11, 2001, in 
an effort to respond quickly to potential threats to aviation security by 
identifying individuals who may pose a threat to aviation security, 
including terrorists planning or executing an attack who were not likely to 
be identified by TSA's other screening security measures. Because TSA 
did not ensure that SPOT'S underlying methodology and work methods 
were scientifically validated prior to its nationwide deployment, an 
independent panel of experts could help determine whether a scientific 
foundation exists for the way in which the SPOT program uses behavior 
detection analysis for counterterrorism purposes in the aviation 
environment. 



Conclusions 



"'The training gap analysis identifies gaps in the training curriculum. 
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With approximately $5.2 billion devoted to screening passengers and their 
property in fiscal year 2009, it is important that TSA provides effective 
stewardship of taxpayer funds ensuring a return on investment for each 
layer of its security system. As one layer of aviation security, the SPOT 
program has an estimated projected cost of about $1.2 billion over the 
next 5 years if the administration's requested funding of $232 million for 
fiscal year 2011 remains at this level. 112 The nation's constrained fiscal 
environment makes it imperative that careful choices be made regarding 
which investments to pursue and which to discontinue. If an independent 
expert panel determines that DHS's study is sufficiently comprehensive to 
determine whether the SPOT program is based on valid scientific 
principles that can be effectively applied in an airport environment for 
counterterrorism purposes, then conducting a comprehensive risk 
assessment including threat, vulnerability, and consequence could 
strengthen TSA's ability in making resource allocation decisions and 
prioritizing its risk mitigation efforts. Moreover, conducting a cost-benefit 
analysis could help TSA determine whether SPOT provides benefits 
greater than or equal to other security alternatives and whether its level of 
investment in the SPOT program is appropriate. Revising its strategic plan 
for SPOT to incorporate risk assessment information, cost and resource 
analysis, and other essential components could enhance the plan's 
usefulness to TSA in making program management and resource 
allocation decisions to effectively manage the deployment of SPOT. 

Providing guidance on how to use TSA's resources for running passenger 
names against intelligence and criminal databases available to the 
Transportation Security Operations Center and helping DHS to connect 
disparate pieces of information using the Transportation Information 
Sharing System and other related intelligence and crime database and data 
sources could better inform DHS and TSA regarding the identity and 
background of certain individuals and thereby enhance aviation security. 
In addition, implementing the steps called for in the TSA Office of 
Strategic Operations plan to provide BDOs with a real-time mechanism to 
verify passenger identities and backgrounds via TSA's Transportation 
Security Operations Center could strengthen their ability to rapidly verify 
the identity and background of passengers who have caused concern, and 
increase the likelihood of detecting and disrupting potential terrorists 



U2 This estimate assumes that there would be no further increases for SPOT over the next 5 
years above the requested $232 million level for fiscal year 2011. However, to stay even 
with inflation, the allocation would likely increase somewhat each year. 
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intending to cause harm to the aviation system. Additionally, developing 
outcome-oriented performance measures, making improvements to the 
SPOT database, and studying the feasibility of utilizing video recordings of 
individuals as they transited checkpoints and who were later charged with 
or pleaded guilty to terrorism-related offenses, could help TSA evaluate 
the SPOT program, identify potential vulnerabilities, and assess the 
effectiveness of its BDOs. Further, developing a plan for systematic and 
periodic evaluation of the training provided to BDOs along with time 
frames and milestones for its completion could help ensure that the SPOT 
training program is evaluated in accordance with its directives to help 
ensure that the program continues to provide BDOs with the necessary 
tools required to carry out their responsibilities. 



To help ensure that SPOT is based on valid scientific principles that can be 
effectively applied in an airport environment, we recommend that the 
Secretary of Homeland Security convene an independent panel of experts 
to review the methodology of the DHS S&T Directorate study on the SPOT 
program to determine whether the study's methodology is sufficiently 
comprehensive to validate the SPOT program. This assessment should 
include appropriate input from other federal agencies with expertise in 
behavior detection and relevant subject matter experts. 

If this research determines that the SPOT program has a scientifically 
validated basis for using behavior detection for counterterrorism purposes 
in the airport environment, then we recommend that the TSA 
Administrator take the following four actions: 

• Conduct a comprehensive risk assessment to include threat, 
vulnerability, and consequence of airports nationwide to determine the 
effective deployment of SPOT if TSA's ongoing Aviation Modal Risk 
Assessment lacks this information. 

• Perform a cost-benefit analysis of the SPOT program, including a 
comparison of the SPOT program with other security screening 
programs, such as random screening, or already existing security 
measures. 

• Revise and implement the SPOT strategic plan by incorporating risk 
assessment information, identifying cost and resources, linking it to 
other related TSA strategic documents, describing how SPOT is 
integrated and implemented with TSA's other layers of aviation 
security, and providing guidance on how to effectively link the roles, 
responsibilities, and capabilities of federal, state, and local officials 
providing program support. 



Recommendations for 
Executive Action 
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• Study the feasibility of using airport checkpoint-surveillance video 
recordings of individuals transiting checkpoints who were later 
charged with or pleaded guilty to terrorism-related offenses to enhance 
understanding of terrorist behaviors in the airport checkpoint 
environment. 

Concurrent with the DHS S&T Directorate study of SPOT, and an 
independent panel assessment of the soundness of the methodology of the 
S&T study, we recommend that the TSA Administrator take the following 
six actions to ensure the program's effective implementation: 

• To provide additional assurance that TSA utilizes available resources to 
support the goals of deterring, detecting, and preventing security 
threats to the aviation system, TSA should: 

• Provide guidance in the SPOT Standard Operating Procedures or 
other TSA directive to BDOs, or other TSA personnel, on inputting 
data into the Transportation Information Sharing System and set 
milestones and a time frame for deploying Transportation 
Information Sharing System access to SPOT airports so that TSA 
and intelligence community entities have information from all SPOT 
LEO referrals readily available to assist in "connecting the dots" and 
identifying potential terror plots. 

• Implement the steps called for in the TSA Office of Security 
Operations Business plan to develop a standardized process for 
allowing BDOs or other designated airport officials to send 
information to TSA's Transportation Security Operations Center 
about passengers whose behavior indicates that they may pose a 
threat to security, and provide guidance on how designated TSA 
officials are to receive information back from the Transportation 
Security Operations Center. 

• Direct the TSA Transportation Security Operations Center to utilize 
all of the law enforcement and intelligence databases available to it 
when running passenger names, for passengers who have risen to 
the level of a LEO referral. 

• To better measure the effectiveness of the program and evaluate the 
performance of BDOs, TSA should: 

• Establish a plan that includes objectives, milestones, and time 
frames to develop outcome-oriented performance measures to help 
refine the current methods used by Behavior Detection Officers for 
identifying individuals who may pose a risk to the aviation system. 

• Establish controls to help ensure completeness, accuracy, 
authorization, and validity of data collected during SPOT screening. 
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• To help ensure that TSA provides BDOs with the knowledge and skills 
needed to perform their duties, TSA should: 

• Establish time frames and milestones for its plan to systematically 
conduct evaluations of the SPOT training program on a periodic 
basis. 



Agency Comments 
and Our Evaluation 



We provided a draft of our report to DHS and TSA on March 19, 2010, for 
review and comment. On May 3, 2010, DHS provided written comments, 
which are reprinted in appendix II. In commenting on our report, DHS 
stated that it concurred with 10 of our recommendations and identified 
actions taken, planned, or under way to implement them. However, the 
actions DHS reported it plans to take and has underway do not fully 
address the intent of our first recommendation. DHS also concurred in 
principle with an eleventh recommendation stating that it had convened a 
working group to determine the feasibility of implementing it. DHS 
commented on the scientific basis underlying SPOT and on two statements 
in our report that it believed were inaccurate — specifically, DHS disagreed 
with our reliance on a 2008 National Research Council report published 
under the auspices of the National Academy of Sciences on issues related 
to behavior detection, and second, on issues related to unpublished 
research they had cited as a partial validation of some aspects of the SPOT 
program. 113 Finally, DHS commented on our conclusion regarding the use 
of the SPOT referral data. 



Regarding our first recommendation that DHS convene an independent 
panel of experts to review the methodology of DHS's Science and 
Technology Directorate (S&T) study on SPOT, and to include appropriate 
input from other federal agencies with relevant expertise, DHS concurred 
and stated the current process includes an independent review of the 
program that will include input from other federal agencies and relevant 
experts. Although DHS has contracted with the American Institutes for 
Research to conduct its study, it remains unclear who will oversee this 
review and whether they are sufficiently independent from the current 
research process. DHS's response also does not describe how the review 
currently planned is designed to determine whether the study's 
methodology is sufficiently comprehensive to validate the SPOT program. 
As we noted in our report, research on other issues, such as determining 



113 The National Research Council is a component of the National Academy of Sciences, a 
part of a private, nonprofit institution, the National Academies, which provide science, 
technology, and health policy advice under a congressional charter. 
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the number of individuals needed to observe a given number of passengers 
moving at a given rate per day in an airport environment or the duration 
that such observation can be conducted by BDOs before observation 
fatigue affects effectiveness, could provide additional information on the 
extent to which SPOT can be effectively implemented in airports. Dr. Paul 
Ekman, a leading research scientist in the field of behavior detection, told 
us that additional research could help determine the need for periodic 
refresher training since no research has yet determined whether behavior 
detection is easily forgotten or can be potentially degraded with time or 
lack of use. Thus, questions exist as to whether behavior detection 
principles can be reliably and effectively used for counterterrorism 
purposes in airport settings to identify individuals who may pose a risk to 
the aviation system. To help ensure an objective assessment of the study's 
methodology and findings, DHS could benefit from convening an 
independent panel of experts from outside DHS to determine whether the 
study's methodology is sufficiently comprehensive to validate the SPOT 
program. 

DHS also concurred with our second recommendation to conduct a 
comprehensive risk assessment to determine the effective deployment of 
SPOT. DHS stated that TSA's Aviation Modal Risk Assessment is designed 
to evaluate overall transportation security risk, not deployment strategies. 
However, DHS noted that TSA is in the process of conducting an initial 
risk analysis using its risk management analysis tool and plans to update 
this analysis in the future. However, it is not clear from DHS's comments 
how this analysis will incorporate an assessment of TSA's deployment 
strategy for SPOT. 

DHS also concurred with our third recommendation to perform a cost- 
benefit analysis of SPOT. DHS noted that TSA is developing an initial cost- 
benefit analysis and that the flexibility of behavior detection officers 
already suggests that behavior detection is cost-effective. However, it is 
not clear from DHS's comments whether its cost-benefit analysis will 
include a comparison of the SPOT program with other security screening 
programs, such as random screening, or already existing security 
measures as we recommended. Completing its cost-benefit analysis and 
comparing it to other screening programs should help establish whether 
the SPOT program is cost-effective compared to other layers of security. 

With regard to our fourth recommendation to revise and implement the 
SPOT strategic plan using risk assessment information, DHS concurred 
and noted that analysis facilitated by the risk management analysis tool 
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will allow the program to revise the SPOT strategic plan to incorporate the 
elements identified in our recommendation. 

DHS also concurred with our fifth recommendation to study the feasibility 
of using airport checkpoint-surveillance video recordings to enhance its 
understanding of terrorist behaviors. DHS noted that TSA agrees this 
could be a useful tool and is working with DHS's S&T Directorate to utilize 
video case studies of terrorists, if possible. These cases studies could help 
TSA determine what behaviors had been demonstrated by these persons 
convicted of terrorist-related offenses who went through SPOT airports, 
and what could be learned from the observed behaviors. 

DHS concurred with our sixth recommendation that TSA provide guidance 
in the SPOT SOP or other directives to BDOs, or other TSA personnel, on 
how to input data into the Transportation Information Sharing System 
database. DHS stated that the SPOT SOP is undergoing revision, and that 
the revised version will provide guidance directing the input of BDO data 
into the Transportation Information Sharing System. DHS anticipates 
release of the updated SPOT SOP in fiscal year 2010. DHS also agreed that 
TSA should set milestones and a time frame for deploying Transportation 
Information Sharing System access to SPOT airports so that TSA and 
intelligence community entities have information from all SPOT LEO 
referrals readily available to assist in "connecting the dots" and identifying 
potential terror plots. DHS stated that TSA is currently drafting a plan to 
include milestones and a time frame for deploying System access to all 
SPOT airports. 

DHS concurred with our seventh recommendation to develop a 
standardized process to allow BDOs or other designated airport officials to 
send information to TSA's Transportation Security Operations Center 
about passengers whose behavior indicates they may pose a threat to 
security, and to provide guidance on how designated TSA officials are to 
receive information back from the Center. DHS stated that TSA has 
convened a working group to address this recommendation. Moreover, 
TSA is developing a system and procedure for sending and receiving 
information from the Center and stated that it anticipates having a system 
in place later in fiscal year 2010. 

DHS concurred in principle with regard to our eighth recommendation 
that the Transportation Security Operations Center utilize all of the 
databases available to it when conducting checks on passengers who rise 
to the level of a LEO referral against intelligence and criminal databases. 
DHS stated that TSA has convened a working group to address this 
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recommendation. According to DHS, this group will conduct a study 
during fiscal year 2010 to determine the feasibility of fully implementing 
this recommendation. As such, the study is to review the various 
authorities, permissions, and limitations of each of the databases or 
systems cited in our report. DHS stated that access to some of the 
systems, requires more justification than a BDO referral. Further, 
according to DHS, because some of the databases or systems contain 
classified information, TSA will also need to adopt a communication 
strategy to transmit the passenger information between the BDO and 
Transportation Security Operations Center. DHS stated that TSA will 
work on a process to collect the passenger information, verify the 
passenger's identity, through checks of databases, and analyze that 
information to determine if the passenger is the subject of an investigation 
and may pose a risk to aviation security. 

With regard to our ninth recommendation to establish a plan with 
objectives, milestones, and time frames to develop outcome-oriented 
performance measures for BDOs, DHS concurred and stated that TSA 
intends to consult with experts to develop outcome-oriented performance 
measures. 

DHS also concurred with our tenth recommendation to establish controls 
for SPOT data. DHS noted that TSA established additional controls as part 
of the SPOT database migration to TSA's Performance Management 
Information System and is exploring an additional technology solution to 
reduce possible errors. As noted in our report, since these changes to the 
database were not complete at the time of our audit, we could not assess 
whether the problems we identified with the database had been corrected. 

Regarding our eleventh recommendation to establish time frames and 
milestones to systematically evaluate the SPOT training program on a 
periodic basis, DHS concurred and stated that TSA intends to develop 
such a plan following completion of DHS's S&T Directorate's BDO Job 
Task Analysis, and TSA's training gap analysis, which identifies gaps in the 
training curriculum. 

DHS also commented on the scientific basis underlying SPOT. 
Specifically, DHS stated that decades of scientific research has shown the 
SPOT behaviors to be "universal in their manifestation." However, 
according to DHS, its S&T Directorate is examining the extent to which 
behavior indicators are appropriate for screening purposes and lead to 
appropriate and correct security decisions. DHS also commented that the 
results of this work, which is currently underway, will establish a scientific 
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basis of the extent to which the SPOT program instruments and methods 
are valid. Thus, DHS's comments suggest that additional research is 
needed to determine whether these behaviors can be used in an airport 
environment for screening passengers to identify threats to the aviation 
system. 

Moreover, DHS took issue with our use of a report from the National 
Research Council of the National Academy of Sciences stating that we 
improperly relied upon this report. 114 We disagree. DHS questioned the 
findings of the National Research Council report and stated that it lacked 
sufficient information for its conclusions because it principally focused on 
privacy as it relates to data mining and behavioral surveillance and was 
not intended to represent an exhaustive or definitive review of the 
research or operational literature on behavioral screening, including 
recent unpublished DHS, defense, and intelligence community studies. 
DHS also stated that the National Research Council report did not study 
the SPOT program and that the researchers did not conduct interviews 
with SPOT personnel. 

As we noted in our report, although the National Research Council report 
addresses broader issues related to privacy and data mining, a senior 
Council official — and one of the authors of the study — stated that the 
committee included behavior detection as a focus because any behavior 
detection program could have privacy implications. This official added 
that the primary objective of the report was to develop a framework for 
sound decision making for programs, such as SPOT, and help ensure a 
sound scientific and legal basis. According to this official, the National 
Academy of Sciences' Committee on Technical and Privacy Dimensions of 
Information for Terrorism Prevention and Other National Goals — which 
had oversight of the report — was briefed on the SPOT program as part of 



'"National Research Council, Protecting Individual Privacy in the Struggle Against 
Terrorists: A Framework for Assessment (Washington, D.C.: National Academies Press, 
2008). The report's preparation was overseen by the National Academy of Science's 21- 
member Committee on Technical and Privacy Dimensions of Information for Terrorism 
Prevention and Other National Goals. We reviewed the approach used and the information 
provided in this study and found the study to be credible for our purposes. The 
contributors included recognized experts across a variety of fields, including William J. 
Perry, former Secretary of Defense, and Dr. Tara O'Toole, then-CEO and Director of the 
Center for Biosecurity of the University of Pittsburgh Medical Center, Professor of 
Medicine and of Public Health at the University of Pittsburgh. (Dr. O'Toole was 
subsequently nominated and confirmed as the Under Secretary of the DHS Science and 
Technology Directorate.) 
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the study. The Committee also conducted meetings with three experts in 
behavior detection as part of their research. During the course of our 
review, we interviewed three Committee members responsible for 
developing the report's findings, as well as four other behavior detection 
experts, including the three who participated in the National Research 
Council study. Our discussions with these experts corroborated the 
report's findings. Thus, we believe that our use of the Council report was 
an appropriate and a necessary part of our review. 

However, the National Research Council report was only one of many 
sources that we analyzed with regard to the science of behavioral and 
physiological screening, and its applicability to an airport environment. As 
we noted in the description of our methodology, our study included 
interviews with officials from DHS as well as several of its components 
and other U.S. government agencies — each of which use elements of 
behavior detection in their daily work. We also interviewed El Al airline 
officials, a former director of security at Israel's Ben-Gurion airport, and 
seven nationally recognized experts in behavior detection as part of our 
review. Moreover, as we explained in the discussion of our scope and 
methodology, we conducted a survey about the SPOT program of all 118 
Federal Security Directors for all SPOT airports, and conducted site visits 
to 15 SPOT airports. In addition, we analyzed the SPOT referral database, 
to the extent the data permitted, covering a 4-year period and the results 
from 2 billion passengers passing through SPOT airports. Moreover, we 
attended both the basic and advanced training courses in behavior 
detection provided by TSA to BDOs, in order to better understand how the 
program is carried out. Therefore, our analysis of the program was not 
derived from or based on a single study by the National Research Council 
as DHS suggested, but rather is based on all of the information we 
gathered and synthesized from multiple, diverse, expert sources, each of 
which provided different perspectives about the program, as well as about 
behavior detection in general. 

DHS also disagreed with the accuracy of a statement included in our 
report that noted DHS S&T could not provide us with specific contacts 
related to sources of information for certain research it cited as support 
for the SPOT program. In its comments, DHS stated that it had provided 
us with all requested documents that represent DHS's S&T Directorate- 
sponsored research. We agree. However, DHS did not provide us with 
contact information for the sources of unpublished studies by the 
Department of Defense and other intelligence community studies that DHS 
S&T had cited as support for the SPOT program. Without such 
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information, we are unable to verify the contents of these unpublished 
studies. 

Finally, DHS stated that while we were unable to use the SPOT referral 
data to assess whether any behavior or combination of SPOT behaviors 
could be used to reliably predict the final outcome of an incident involving 
the use of SPOT, it was able to analyze the SPOT referral database 
successfully after working with TSA to verify scores assigned to different 
indicators. Our concern with the data did not involve the question of 
whether some behaviors were entered erroneously, nor whether errors in 
coding were excessive or non-random. Rather, we were concerned with 
whether the data on behaviors were complete. Specifically, it cannot be 
determined from the SPOT referral database whether all behaviors 
observed were included for each referred passenger by each BDO or 
whether only the behaviors that were sufficient for a LEO referral were 
recorded into the database. It is not possible to determine from the 
database if the number of observed behaviors entered for a given 
passenger was the total number of observed behaviors, or whether 
additional behaviors were observed. A rigorous analysis of the relative 
effects of the different behaviors on the outcomes of the use of SPOT 
would require each BDO to record, for each of the observable behaviors, 
whether it was or was not observed. 

TSA also provided technical comments that we incorporated as 
appropriate. 



We will send copies of this report to the Secretary of Homeland Security; 
the TSA Administrator (Acting); and interested congressional committees 
as appropriate. The report will also be available at no charge on the GAO 
Web site at http://www.gao.gov. 

If you or your staff have any questions about this report, please contact me 
at (202) 512-4379 or lords@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last page 
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of this report. Key contributors to this report are acknowledged in 
appendix III. 

Sincerely yours, 




Stephen M. Lord 

Director, Homeland Security and Justice Issues 
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Appendix I: Scope and Methodology 



To determine the extent to which the Transportation Security 
Administration (TSA) determined whether the Screening of Passengers By 
Observation Techniques (SPOT) program had a scientifically-validated 
basis for identifying passengers before deploying it, we reviewed literature 
on behavior analysis by subject matter experts, interviewed seven experts 
in behavior analysis, interviewed other federal agencies and entities about 
how they use behavior detection techniques, and analyzed relevant reports 
and books on the topic. These included a 2008 study by the National 
Research Council of the National Academy of Sciences that has a 
discussion regarding deception and behavioral surveillance, as well as 
other issues related to behavioral analysis. 1 We interviewed Dr. Herbert S. 
Lin, who was a primary author of the report, as well as Dr. Robert W. 
Levenson, and Dr. Stephen E. Fienberg, both members of the Academy 
committee that oversaw the report, about the report's findings with regard 
to behavior detection, and the extent to which behavior detection in a 
complex environment, such as an airport terminal, has been validated with 
regard to its effectiveness in identifying persons who may be a risk to 
aviation security. Other behavior detection experts we consulted were Dr. 
Paul Ekman; 2 Dr. Mark Frank; 3 Dr. David Givens; 4 Dr. David Matsumoto; 5 



'National Research Council, Protecting Individual Privacy in the Struggle Against 
Terrorists: A Framework for Assessment (Washington, D.C.: National Academies Press, 
2008). The report's preparation was overseen by the NAS's 21-member Committee on 
Technical and Privacy Dimensions of Information for Terrorism Prevention and Other 
National Goals. We reviewed the approach used and the information provided in this study 
and found the study to be credible for our purposes. The contributors included recognized 
experts across a variety of fields, including William J. Perry, former Secretary of Defense, 
and Dr. Tara O'Toole, then-CEO and Director of the Center for Biosecurity of the 
University of Pittsburgh Medical Center, Professor of Medicine and of Public Health at the 
University of Pittsburgh. (Dr. O'Toole was subsequently nominated and confirmed as the 
Under Secretary of DHS's Science and Technology Directorate. The National Research 
Council is a component of the National Academy of Sciences, a part of a private, nonprofit 
institution, the National Academies, which provide science, technology, and health policy 
advice under a congressional charter 

2 Dr. Ekman is professor emeritus of psychology at the University of California Medical 
School, San Francisco, and is considered one of the world's foremost experts on facial 
expressions. His books include: Emotions Revealed: Recognizing Faces and Feelings to 
Improve Communications and Emotional Life (New York: Holt and Company, 2003); 
Emotion in the Human Face (New York: Pergamon Press, 1972); Unmasking the Face: A 
guide to Recognizing Emotions from Facial Clues (Englewood Cliffs, N.J.: Prentice-Hall, 
1975). Dr. Ekman has published more than 100 articles. 

3 Dr. Frank is Associate Professor, Department of Communication, College of Arts and 
Sciences, at the University at Buffalo, State University of New York. He is on the Advisory 
Board of the University's Center for Unified Biometrics and Sensors, and has conducted 
research supported by DHS, the Defense Advanced Research Projects Agency, and the 
National Science Foundation. 
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and Mr. Rafi Ron, former director of security at Israel's Ben-Gurion 
Airport. Dr. Ekman, Dr. Frank, and Mr. Ron provided expert advice for the 
National Research Council study. Dr. Givens was identified by TSA as 
having been their principal source for the nonverbal behavior indicators 
used by the SPOT program. We also interviewed Dr. Lawrence M. Wein, 
an expert in emergency responses to terror attacks and mathematical 
models in operations management. 6 In addition, we interviewed officials 
from the Department of Homeland Security's (DHS) Science and 
Technology (S&T) Directorate regarding their ongoing research into 
behavior detection. Although the views of these experts cannot be 
generalized across all experts in behavior analysis because we selected 
individuals based on their publications on behavioral analysis or related 
topics, their recognized accomplishments and expertise, and, in some 
cases, TSA's use of their work or expertise to design and review the SPOT 
program's behaviors, they provided us with an overall understanding of 
the fundamentals of behavior analysis, and how it could be applied. 

To determine the basis for TSA's strategy to develop and deploy SPOT and 
evaluate to what extent SPOT was informed by a cost-benefit analysis and 
a strategic plan, we reviewed program documentation, including briefings 
prepared by the SPOT program office during the course of developing and 
fielding SPOT, two versions of a strategic plan for SPOT, and the 2009 
SPOT standard operating procedures guidance. We compared the plans 
and analyses used by TSA to develop and implement SPOT to criteria on 
how to develop and implement programs in DHS's 2006 Cost Benefit 
Analysis Guidebook, 7 as well as to Office of Management and Budget 
guidance on the utility of cost-benefit analyses in program 



4 Dr. Givens is the director of the nonprofit Center for Nonverbal Studies, in Spokane, 
Washington. He is the author of Love Signals: A Practical Field Guide to the Body 
Language of Courtship (St. Martin's, New York, 2005) and Crime Signals: How to Spot a 
Criminal Before You Become a Victim (St. Martin's, 2008). The Center's Web site links to 
Dr. Givens' reference tool, The Nonverbal Dictionary of Gestures, Signs and Body 
Language Cues. 

5 Dr. Matsumoto is a Professor, Department of Psychology at San Francisco State 
University, and is an associate of Dr. Ekman. 

6 Dr. Wein is the Paul E. Holden Professor of Management Science at the Graduate School 
of Business, Stanford University. His homeland security-related work includes four papers 
in Proceedings of the National Academy of Sciences, on an emergency response to a 
smallpox attack, an emergency response to an anthrax attack, a biometric analysis of the 
US-VTSIT Program, and an analysis of a bioterror attack on the milk supply. 

7 DHS, Cost Benefit Analysis Guidebook (Washington, D.C.: February 2006). 
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implementation. We also analyzed the development of SPOT in light of 
the standards and criteria cited in DHS's 2006 National Infrastructure 
Protection Plan. We met with relevant TSA officials to discuss these 
issues. To assess whether DHS developed an effective strategic plan for 
SPOT prior to implementing the program, we interviewed TSA officials 
involved in development of the SPOT strategic plan. We analyzed whether 
the SPOT plan incorporated the desirable characteristics of an effective 
strategic plan as identified by previous GAO work on what strategic plans 
should include to be considered effective, such as a risk assessment, cost 
and resources analysis, and a means for collaboration with other key 
entities. 9 We also examined it in light of the requirements of the 
Government Performance and Results Act of 1993, which specifies the 
elements of strategic plans for government programs. 10 We assessed 
whether the SPOT strategic plan was followed by TSA. As part of our 
analysis of the planning for SPOT before it was implemented on a 
nationwide basis, we reviewed TSA documentation related to the 
development and pilot testing of SPOT, such as a TSA white paper on 
SPOT, and interviewed key program officials from both headquarters and 
field offices. 11 

We also interviewed cognizant officials from other U.S. government 
agencies and agency entities that utilize behavior detection in their work, 
including U.S. Customs and Border Protection (CBP), the U.S. Secret 
Service, the TSA's Federal Air Marshal Service (FAMS) component, and 
the Federal Bureau of Investigation (FBI). We sought their views on the 
utility of various behavior detection methods, their experience with 
practicing behavior detection, and asked them about the extent to which 
TSA had consulted with them in developing and implementing the SPOT 
program. 

To better understand how SPOT incorporated expertise about the use of 
behavior detection in an airport setting, we interviewed officials from 



Office of Management and Budget (OMB), Circular No. A-94, Guidelines and Discount 
Rates for Benefit-Cost Analysis of Federal Programs (Washington, D.C.: October 1992); 
and Circular A-4, Regulatory Analysis (Washington, D.C.: September 2003). 

9 GAO-04-408T. 

10 Pub. L. No. 103-62, 107 Stat. 285 (1993). 

"TSA, Screening of Passengers by Observation Technique (SPOT) White Paper for the 
Department of Homeland Security (Washington, D.C.: Feb. 22, 2005). 
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Israel's El Al Airlines, which is cited by TSA as having provided part of the 
basis of the SPOT program. We asked about El Al's methods to ensure the 
security of its passenger aircraft, and also interviewed a former head of 
security at Israel's Ben-Gurion airport, who has advised TSA on security 
issues. We asked TSA and SPOT program officials about their 
consultations with El Al, and about the ways in which they had utilized El 
Al's expertise, as well as about any other entities whose expertise they 
may have adopted into SPOT. 

To determine the challenges, if any, that emerged during implementation 
of the SPOT program, we interviewed headquarters and field personnel 
about how the program has utilized the resources available to it to ensure 
that it is effective. These resources included the support of law 
enforcement officers (LEOs), to whom passengers are referred by 
Behavior Detection Officers (BDOs) for additional questioning. In 
addition, we interviewed SPOT program and TSA officials about the 
databases available to them at TSA's Transportation Security Operations 
Center to determine if a suspect passenger is being sought by other U.S. 
law enforcement or intelligence entities, and whether there is guidance for 
BDOs on when and how to contact the Transportation Security Operations 
Center. We also asked about whether there is guidance and training for 
BDOs on how to access TSA's Transportation Information Sharing System 
database, which is owned by FAMS, and is available through the 
Transportation Security Operations Center. 12 To determine if any 
management challenges had emerged related to management controls in 
developing and implementing SPOT, we compared TSA's approach for 
implementing and managing the SPOT program with GAO's Standards for 
Internal Control in the Federal Government 13 and with risk management 
principles we had previously identified. 14 Our legal counsel office 
reviewed court decisions relevant to the SPOT program. In addition, we 
interviewed attorneys from the American Civil Liberties Union, and 
obtained and reviewed TSA's Privacy Impact Assessments for SPOT, the 
Transportation Security Operations Center, and the Transportation 



12 The data from interviews of suspicious passengers by FAMS are inputted into the 
Transportation Information Sharing System, as are reports sent to FAMS from airline 
employees about suspicious passengers. 

13 GAO/AIMD-00-21.3.1. 

14 GAO, Transportation Security: Comprehensive Risk Assessments and Stronger Internal 
Controls Needed to Help Inform TSA Resource Allocation, GAO-09-492 (Washington, D.C.: 
Mar. 27, 2009). 
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Information Sharing System. We also met with and discussed relevant 
privacy and legal issues with TSA's Offices of Privacy and Civil Rights/Civil 
Liberties. To obtain data about certain aspects of the SPOT program that 
the SPOT program office did not have, we conducted a survey of Federal 
Security Directors 15 whose responsibilities included security at all 161 
SPOT airports at the time of our survey. (Some Federal Security Directors 
have responsibility for more than one airport.) We obtained a 100 percent 
response rate. This survey asked, among other things, about whether 
there were cameras at security checkpoints that record the interactions of 
Transportation Security Officers (TSO), BDOs, and passengers; if the 
airport authority had an agreement with TSA that specifies certain law 
enforcement actions during a SPOT referral; and if there was an 
agreement, or any other comparable guidance that specified a time limit 
for LEOs to come to checkpoints after being called for help by BDOs. 

To determine the extent to which TSA has measured SPOT's effect on 
aviation security, we obtained and analyzed the TSA SPOT referral 
database, which records all incidents in which BDOs refer passengers to 
secondary, more intensive questioning, and which also records all 
incidents in which BDOs chose to refer passengers to LEOs. We found 
that the SPOT database was sufficiently reliable to count the number of 
arrests resulting from referrals from BDOs to LEOs, for examining the 
reasons for each arrest, and for counting the percentage of times that 
LEOs responded to BDO calls for service, and the length of time required. 
Use of these data required us to resolve apparent contradictions and 
anomalies in the database to make the data useable. Because of data 
problems, we were unable to conduct analyses to assess whether any 
behavior or combination of behaviors could be used to predict the final 
outcome of an incident involving the use of SPOT. In addition, we 
reviewed relevant standardization team reports and observed a 
standardization team visit in operation. 

In addition, we spoke with BDO managers, Federal Security Directors, and 
Assistant Federal Security Directors to determine how BDOs are 
evaluated. To do so, we conducted site visits to 15 commercial airports at 
which BDOs and SPOT have been deployed, or almost 10 percent of the 



"Federal Security Directors are the highest ranking TSA security officials at U.S. airports; 
Assistant Federal Security Directors are their assistants. Both are responsible for all 
aspects of security at airports, including coordination with federal and nonfederal law 
enforcement entities operating at airports, such as FAMS, the Drug Enforcement 
Administration, and CBP. 
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161 airports with SPOT. We chose these airports taking into account the 
following criteria, among others: (1) each airport had BDOs deployed, and 
at each, the SPOT program had been in effect for no less than 3 months; 
(2) airports were chosen to provide a variety of sizes, as measured in 
annual passenger volume; physical location within the country (northeast, 
southwest, central, Pacific Coast, rural, urban); and estimated risk of 
terrorist incident, using DHS's Current Airports Threat Assessment 16 list 
(visiting 6 that were in the top 10, and others much lower); (3) BDOs who 
are employed by contractors, rather than employed directly by TSA; and 
(4) airports with LEOs who were identified to us by TSA as having 
received some form of behavior detection training and airports where they 
were not known to have received such training. In addition, we took into 
account the location of the airports with regard their proximity to subject 
matter experts on behavior detection whom we wished to interview, as 
well as the time and cost required to reach certain airports. 

At each of the airports we visited, we interviewed cognizant officials, 
including the Federal Security Director or Assistant assigned to the 
airport, the BDO program manager, one or two BDOs, and one or two 
LEOs who have interacted with BDOs. Since each of these airports differs 
in terms of passenger volume, physical size and layout, geographic 
location, and potential value as a target for terrorism, among other things, 
the results from these visits are not generalizable to other airports. 
However, these visits provided helpful insight into the operation of SPOT 
at airports. 

In addition, to determine if individuals had transited SPOT airports who 
were later charged with or pleaded guilty to terrorism-related offenses, we 
reviewed information contained in (1) the Treasury Enforcement 
Communication System II database maintained by CBP; 17 (2) Department 
of Justice information and court documents, including indictments and 
related documents; and (3) media accounts of individuals accused of 



1G The Current Airports Threat Assessment is a threat estimate designed to provide a 
snapshot of the current terrorist threat to airports in the United States as well as for major 
international airports serving as last points of departure for U.S. airlines. 

"The Treasury Enforcement Communication System was designed to provide controlled 
access to a large database of information about suspects and to interface with a number of 
other law enforcement systems. These capabilities are provided to users through various 
applications, including the Inspection/Interagency Border Inspection System applications 
that facilitate passenger processing through the implementation of innovative border 
control technology. 
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terrorism-related activities. We compared information pertaining to these 
individuals' dates of transit to the dates when SPOT was deployed to the 
various airports identified in the Treasury Enforcement Communication 
System and Justice Department data to determine if SPOT had been 
deployed at a given airport when the transits occurred. Further, we used 
our survey of Federal Security Directors at SPOT airports to determine the 
extent to which video surveillance cameras are present at checkpoints. 

To assess the extent that SPOT training incorporates the attributes of an 
effective training program, we had training experts at TSA headquarters 
complete a training assessment tool that we developed using our prior 
work for assessing training courses and curricula. 18 To address training- 
related issues, including to understand better how other entities train their 
employees in behavior detection, and what their curricula include, we 
conducted site visits to the Secret Service, FAMS, CBP, and the FBI, and 
also interviewed nongovernmental experts on behavior detection (our 
selection of these experts is discussed above). As part of our assessment 
of SPOT training, we attended the basic SPOT training course given to 
BDOs, as well as the advanced SPOT course on behavior detection. We 
interviewed BDOs and BDO managers about the SPOT training, as well as 
officials of El Al airlines, with regard to how El Al trains and tests its 
personnel who utilize behavior recognition and analysis as part of their 
assessment of El Al passengers. 

We conducted this performance audit from May 2008 through May 2010, in 
accordance with generally accepted government auditing standards. 
Those standards require that we plan and perform the audit to obtain 
sufficient, appropriate evidence to provide a reasonable basis for our 
findings and conclusions based on our audit objectives. We believe that 
the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 



GAO, Human Capital: A Guide for Assessing Strategic Training and Development 
Efforts in the Federal Government, GAO-04-546G (Washington, D.C.: Mar. 1, 2004). 
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US. Department of Homeland Security 

Washington. DC 20528 

Homeland 
Security 

May 3, 2010 



Mr. Steve Lord 

Director, Homeland Security & Justice Issues 
U.S. Government Accountability Office (GAO) 
44 1G Street, NW 
Washington, DC 20548 

Dear Mr. Lord: 

Thank you for the opportunity to review and comment on GAO-10-157SU, the draft report 
titled: Aviation Security: Efforts to Validate Aspects o/TSA 's Screening of Passengers by 
Observation Techniques (SPOT) Program Underway, But Opportunities Exist to Strengthen 
Validation and Address Operational Changes. The Transportation Security Administration 
(TSA) appreciates the U.S. Government Accountability Office's work in planning and 
conducting its review and issuing this report. 

TSA deployed the SPOT program in an effort to mitigate the threat of individuals with 
potentially hostile intent from boarding a commercial airplane and causing harm. Congress has 
encouraged the use of behavior recognition to enhance aviation security and has provided 
resources to support its implementation and expansion. The SPOT program fulfills the mandate 
of Section 161 1 of the Implementing Recommendations of the 9/1 1 Commission Act, P.L. 1 10- 
53, that "TSA shall provide advanced training to the transportation security officers for the 
development of specialized security skills, including behavior observation and analysis ... in 
order to enhance the effectiveness of layered transportation security measures." 

Intelligence continues to show there is no specific terrorist profile. In a March 10, 2010, 
hearing before the Senate Homeland Security and Governmental Affairs Committee, TSA Acting 
Administrator Gale Rossides highlighted the challenge faced by TSA leaders in "balancing the 
requirement to screen all passengers and to actually focus our officers' attention on the right 
passengers." TSA designed SPOT to increase its ability to focus on the "right passengers" by 
identifying persons exhibiting behaviors and appearances that may indicate stress, fear, and 
deception, and distinguish them from other travelers. 

SPOT is Based on Scientific Research and Law Enforcement Practices 

TSA's development and deployment of SPOT was a planned and deliberate process based on 
more than 3 years of operational test-bed assessment of SPOT at Boston's Logan International 
Airport from June 2003 until nationwide rollout began in fiscal year (FY) 2007. TSA carefully 
developed SPOT by using selective behaviors recognized within both the scientific and law 




www.dhs.gov 
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enforcement communities as displaying stress, fear, and deception. A SPOT working group, 
made up of various TSA and U.S. Department of Homeland Security (DHS) components, 1 was 
created in February 2004. Other organizations, such as the Massachusetts State Police, the 
Federal Bureau of Investigation (FBI) Behavioral Sciences Unit, and the Federal Law 
Enforcement Training Center, were also involved in SPOT development Through these 
working groups, TSA has developed and finalized SPOT standard operating procedures (SOPs) 
for a common ability to assess behaviors indicating hostile intent for both aviation and mass 
transit modes of transportation. TSA continues to consult with its SPOT working group partners 
as it updates the procedures and science behind the program. 

Decades of scientific research have shown the behaviors to be universal in their 
manifestation. In fact, the DHS Science and Technology Directorate (S&T) completed a study 
on suicide bomber indicators in July 2009 that illustrates a very high degree of overlap between 
operationally reported suicide bomber indicators and TSA SPOT behaviors. This result further 
bolsters TSA's contention that the SPOT program draws fiom the best practices of many defense, 
intelligence, and law enforcement organizations. 

SPOT Scientific Validation is Ongoing 

S&T began research in 2007 to examine the validity of the SPOT program. The series of 
studies involved in this research is designed to assess the validity of the SPOT scoring system, 
including the use of individual behavioral indicators to identify high-risk travelers. More 
specifically, S&T's research plan aims to examine the extent to which these behavioral indicators 
are appropriate for screening purposes and lead to appropriate and correct security decisions. 
When this study is complete, SPOT will be one of the most, if not the most, rigorously tested 
behavior-based security screening programs in existence. 

Results of this work will establish a scientific basis of the extent to which the SPOT program, 
including its instrument and methods, such as the SPOT Referral Report and SOPs, are valid. 
Although it is challenging to establish the validity of a deterrent program in which the outcomes 
of interest are extremely rare, critical elements of reliability and validity will be rigorously 
assessed. Of particular importance is the evaluation of criterion-related validity, or the extent to 
which travelers are correctly selected for screening based on the SPOT scoring system. 
Establishing this degree of classification accuracy justifies the use of the SPOT program to 
discriminate high-risk travelers from low-risk travelers. Regardless of any other metrics, the 
extent to which the SPOT scores accurately identify high-risk travelers is critically important to 
program validity. 

Following criterion-related validity, the next central element of validity is the consistency of 
implementation of the instrument and program. This will be examined in a variety of ways, 
including an investigation of the consistency in the operational use of SPOT behavioral 
indicators Behavior Detection Officers (BDOs) and across locations and time periods, all of 
which represents reliability assessment. Finally, construct-related validity, or the extent to which 



' Includes TSA's Office of Civil Rights, Office of Chief Counsel, and Privacy Office, and 
DHS's Policy Office and Transportation Security Laboratory. 
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the SPOT program behaviors truly represent the expressions of high-risk travelers, will be 
examined by comparing the SPOT behaviors to similar instruments in use for the same purpose. 
S&T's July 2009 study of suicide bomber indicators was the first step in evaluating construct- 
related validity. 

This research is expected to be completed in FY 201 1 . TSA understands that after this 
validation is complete, there will be other areas where further research should be conducted, and 
it is TSA's intention to complete this research. 

National Academy of Sciences (NAS) Report Does Not Represent an Exhaustive or 
Definitive Review of the Research or Operational Literature on Behavioral Screening 

TSA would like to specifically address a few comments in the GAO-10-157SU report that we 
believe are inaccurate. The report draws heavily from a National Academy of Sciences (NAS) 
report which is being improperly relied upon. As the sponsor of the NAS study, DHS S&T 
questioned its findings, stating that the study lacked sufficient information for its conclusions 
because the NAS study principally focused on privacy as it relates to behavioral surveillance— not 
on behavioral surveillance technology itself. The study was not intended to, and the results do 
not represent an exhaustive or definitive review of the research or operational literature on 
behavioral and physiological screening, including recent findings from unpublished DHS, 
defense, and intelligence community studies. Furthermore, it should be noted that the report did 
not study the SPOT program, nor did any of the researchers conduct interviews with SPOT 
program personnel. 

Additionally, GAO states that "DHS S&T could not provide us with specific contacts related 
to the sources of this research." This statement is not accurate. The record should reflect that 
DHS S&T provided all requested documents that represented S&T-sponsored research and for 
which S&T possessed the requisite release authority. DHS was not able to release specific 
documents related to research for which it was not the originator. 

The report further states that the audit team was unable to use the SPOT referral data to assess 
whether any behavior or combination of SPOT behaviors could be used to reliably predict the 
final outcome of an incident involving the use of SPOT. However, DHS S&T was able to 
successfully conduct some preliminary analysis of the SPOT referral database. Prior to analysis 
of the SPOT reports, S&T worked with TSA to verify the scores assigned to each indicator with 
the SPOT score sheets and to rescore the pertinent sections and total accordingly for nearly 
100,000 operational reports from 2008. While random errors were noted, errors in large 
databases that require manual entry are not uncommon. Convention suggests that large 
databases like this typically include an error rate of 3 to 5 percent. As long as such errors are 
random, the analytical method is robust enough to account for random errors in this range. 

In conclusion, TSA strongly believes that behavior detection is a vital layer in its aviation 
security strategy, and will continue to strengthen as the program matures. Leaders within the 
community of behavior detection researchers agree. TSA appreciates GAO's work to identify 
opportunities to enhance the SPOT program, and we will continue to work diligently to address 
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the issues identified by GAO. Our ongoing progress demonstrates our commitment to TSA's 
mission of securing our Nation's transportation systems. 

We also appreciate the opportunity to provide you with, in collaboration with DHS S&T, 
comments to GAO's audit recommendations. 

Recommendation 1: To help ensure that SPOT is based on valid scientific principles that 
can be effectively applied in an airport environment, we (GAO) recommend that the 
Secretary of Homeland Security convene an independent panel of experts to review the 
methodology of the S&T Directorate study on the SPOT program before the study is 
implemented to determine whether the study's methodology is sufficiently comprehensive 
to validate the SPOT program. This assessment should include appropriate input from 
other federal agencies with expertise in behavior detection and relevant subject matter 
experts. 

Concur. The U.S. Department of Homeland Security (DHS) Science & Technology 
Directorate's (S&T) current validation process includes an independent and comprehensive 
review of the ongoing SPOT study to be conducted in support of and in collaboration with the 
TSA SPOT program. The assessment will include input from other Federal agencies with 
expertise in behavior detection and relevant subject matter experts. S&T will work with TSA to 
present the SPOT validation project to the panel, produce a report summarizing the panel's 
recommendations, and implement pertinent suggestions in FY 201 0. 

GAO further recommends that if this research determines that the SPOT program has a 
scientifically validated basis for using behavior detection for counterterrorism purposes in 
the airport environment, then the TSA Administrator take the following actions: 

Recommendation 2: Conduct a comprehensive risk assessment to include threat, 
vulnerability, and consequence of airports nationwide to determine the effective 
deployment of SPOT if TSA's ongoing Aviation Modal Risk Assessment lacks this 
information. 

Concur. TSA's Aviation Modal Risk Assessment (AMRA) is designed to evaluate the 
transportation security risk landscape and compare it to other modes. However the AMRA does 
not evaluate risk effectiveness of countermeasures or optimal deployment strategies. For the 
Aviation mode, TSA uses the Risk Management Analysis Tool (RMAT), a risk simulation model 
based on laboratory and operational data that evaluates risk using threat inputs, vulnerability 
information, and consequence estimates. TSA is in the process of conducting an initial risk 
analysis on the SPOT program using RMAT. The risk analysis is based on the initial SPOT 
validation results and will be updated as the validation study results are finalized. 

Recommendation 3: Perform a cost-benefit analysis of the SPOT program including a 
comparison of the SPOT program with other security screening programs, such as random 
screening, or already existing security measures. 

Concur. The SPOT program will use RMAT to perform a cost-benefit analysis of Behavior 
Detection Officers (BDOs) as a countermeasure. The first step in the process is the initial risk 
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assessment that is being conducted on the SPOT program using RMAT. For the cost-benefit 
analysis, costs will be defined as the 5-year total cost of the countermeasure across the aviation 
system. Benefit will be defined as risk-reduction across the aviation security system against a 
portfolio of scenarios. TSA is currently developing an initial cost-benefit analysis for a variety 
of passenger-screening countermeasures including BDOs using die RMAT tool as a basis for 
analysis. BDOs' flexibility across a variety of risk scenarios suggests that behavior detection is a 
cost-effective countermeasure. 

Recommendation 4. Revise and implement the SPOT strategic plan by incorporating risk 
assessment information, identifying cost and resources, linking it to other related TSA 
strategic documents, describing how SPOT is integrated and implemented with TSA's 
other layers of aviation security, and providing guidance on how to effectively link the 
roles, responsibilities and capabilities of federal, state, and local officials providing 
program support 

Concur. The RMAT risk analysis of the BDO program is assisting the SPOT program in 
identifying other countermeasure capabilities that are linked to the behavior detection capability. 
This analysis will allow the SPOT program to develop a revision to the SPOT strategic plan that 
will incorporate the elements identified in the recommendation. 

Recommendation 5: Study the feasibility of using airport checkpoint-surveillance video 
recordings of individuals transiting checkpoints, and who were later charged with or 
pleaded guilty to terrorism-related offenses, to enhance its understanding of terrorist 
behaviors in the airport checkpoint environment. 

Concur. TSA will study the feasibility of using checkpoint surveillance video recordings of 
individuals transiting checkpoints, and who were later charged with or pleaded guilty to 
terrorism-related offenses. TSA agrees that this could be a useful tool in understanding terrorist 
behaviors in the checkpoint environment. 

Additionally, TSA is currently working with DHS S&T/Human Factors to conduct operational 
video validation of the SPOT program. TSA will use a variety of video case studies to validate 
the SPOT program including, if possible, reviewing video of terrorists transiting the TSA 
checkpoint. It is exceedingly rare, however, for video cameras to capture terrorists transiting 
TSA checkpoints. Unfortunately, this factor significantly reduces the feasibility of conducting 
these case studies. 

GAO also recommends that concurrent with the DHS S&T Directorate study of SPOT, and 
an independent panel assessment of the soundness of the methodology of the S&T study, 
the TSA Administrator take the following actions: 

Recommendation 6: Provide guidance in the SPOT SOP or other TSA directive to BDOs, 
or other TSA personnel, on inputting data into the Transportation Information Sharing 
System (TISS) and set milestones and a timeframe for deploying Transportation 
Information Sharing System access to SPOT airports so that TSA and intelligence 
community entities have information from all SPOT Law Enforcement officer (LEO) 
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referrals readily available to assist in "connecting the dots" and identifying potential terror 
plots. 

Concur. TSA is currently undergoing a revision of the SPOT Standard Operating Procedure 
(SOP). The SOP will provide guidance directing the input of BDO data into T1SS. TSA 
anticipates release of the updated SPOT SOP in FY 2010. Additionally, TSA is currently drafting 
a formal plan to include milestones and a timeframe for deploying T1SS access to all SPOT 
airports. 

Recommendation 7: Implement the steps called for in the TSA Office of Security 
Operations Business plan to develop a standardized process for allowing BDOs or other 
designated airport officials to send information to TSA's Transportation Security 
Operations Center (TSOC) about passengers whose behavior indicates that they may pose 
a threat to security, and provide guidance on how designated TSA officials are to receive 
information back from the Transportation Security Operations Center. 

Concur. TSA has convened a working group made up of members of the Office of Security 
Operations, Office of Chief Counsel, Office of Intelligence, and the Office of Law 
Enforcement/Federal Air Marshal Service (FAMS) to address this recommendation. TSA is 
developing a system and procedure for sending and receiving information from the TSOC and 
anticipates having a system in place in FY 2010. It should be noted that information from BDO 
referrals has been transmitted to the TSOC previously; however, TSA agrees to institute a 
standardized process. 

Recommendation 8: Utilize all of the databases available to the Transportation Security 
Operations Center when running passengers who rise to the level of a LEO referral against 
intelligence and criminal databases. 

Concur in principle. TSA has convened a working group composed of members of the Office 
of Security Operations, Office of Chief Counsel, Office of Intelligence, and the Office of Law 
Enforcement^ AMS to address this recommendation. This group will conduct a feasibility study 
during FY 201 to examine if this recommendation can be fully implemented. This study will 
look at the various authorities, permissions, and limitations of each of the databases or systems 
cited in this audit Access to some of the systems, such as Criminal History Record Check 
(CHRC), requires more justification than a BDO referral. Because some of the databases or 
systems contain classified information, TSA will also need to adopt a communication strategy to 
transmit the passenger information back and forth between the BDO and TSOC. TSA will work 
on a process to collect the passenger information, verify the passenger's identity, through checks 
of databases, and analyze that information to determine if the passenger is the subject of an 
investigation and may pose a risk to aviation. 

Recommendation 9: Establish a plan that includes objectives, milestones, and timeframes 
to develop outcome-oriented performance measures to help refine the current methods 
used by Behavior Detection Officers for identifying individuals who may pose a risk to the 
aviation system. 
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Concur. TSA understands the value of outcome-oriented performance measures. However, as 
noted by GAO, there is difficulty in establishing these measures for a deterrence-based program. 
Nonetheless, TSA will consult with industry experts to develop outcome-oriented performance 
measures. TSA will establish a plan that includes objectives, milestones, and timeframes, with 
an end result of producing outcome-oriented performance measures to help refine the current 
methods used by BDOs for identifying individuals who may pose a risk to the aviation system. 

Recommendation 10: Establish controls to help ensure completeness, accuracy, 
authorization, and validity of data collected during SPOT screening. 

Concur. In March 2010, TSA migrated the SPOT database to TSA's Performance Management 
Information System. This migration greatly enhances the SPOT program's capabilities, as they 
relate to completeness, accuracy, authorization, and validity of data collected during SPOT 
screening. Additional controls have been put in place to address the shortcomings of the 
previous database which were highlighted by GAO. TSA is also examining a technology 
solution to allow one-time transcription of all SPOT referral data. This will reduce the 
possibility of errors due to incorrect transcription from one medium to another. 

Recommendation 11: Establish timeframes and milestones for its plan to systematically 
conduct evaluations of the SPOT training program on a periodic basis. 

Concur. DHS S&T, in conjunction with TSA has sponsored a BDO Job Task Analysis (JTA). 
Outputs of the JTA will include Knowledge, Skills, Abilities, and Other characteristics of BDOs 
and Training Learning Objectives. These two items will enable TSA to conduct an in-depth 
training gap analysis. This analysis will begin immediately following completion of the JTA and 
will take approximately three months to complete. Upon completion of the training gap analysis, 
TSA will develop detailed project plans with milestones and schedules based on the scope of the 
overall curriculum development/revision effort. 



Sincerely yours, 




Jerald E. Levine 
Director 

DHS GAO/OIG Liaison Office 
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